# cds-plugin-llm

[![CI](https://github.com/kalyanjanumpally/sap-joule-ai-plugin/actions/workflows/ci.yml/badge.svg)](https://github.com/kalyanjanumpally/sap-joule-ai-plugin/actions/workflows/ci.yml)
[![npm](https://img.shields.io/npm/v/@saptarishi/cds-plugin-llm.svg)](https://www.npmjs.com/package/@saptarishi/cds-plugin-llm)
[![license](https://img.shields.io/npm/l/@saptarishi/cds-plugin-llm.svg)](./LICENSE)

LLM-agnostic AI platform for SAP CAP. **11 providers** behind one unified interface — Anthropic, Google Gemini, AWS Bedrock, Azure OpenAI, Ollama, Groq, Fireworks, DeepSeek, Mistral, any OpenAI-compatible endpoint, and SAP Generative AI Hub — plus **30+ middleware primitives** spanning cost, resilience, security, observability, compliance, RAG, and CI eval.

**Status:** stable (**v2.0.0**). Formal API stability contract in force for the 2.x line — see [`MIGRATION.md`](./MIGRATION.md). 11 providers implemented; **2188 automated tests** on Node 20 + 22 in CI; zero known bugs, zero deprecated APIs; TypeScript definitions ship in the package.

## What it is

A CAP service kind that turns `cds.connect.to('llm')` into a working LLM client — with one unified interface (`chat`, `stream`, `embed`, `batch`) that speaks to any of eleven backends. Swapping backends is a config change, not a code change.

Beyond the provider abstraction, the package ships a comprehensive **middleware platform**: cost + budget guards, cost forecasting, circuit breakers, bulkheads (with latency- and quota-driven adaptive tuners), retries, deadlines, provider fallback, region failover, prompt-injection detection, PII redaction, content-safety classifiers, structured-output validators, response scoring, multi-model consensus voting, prompt-cache metrics, response caches (exact + semantic + pluggable-vector-store), embedding dedup, JSON logs, OTel spans, Prometheus metrics, tenant isolation, distributed locks, compliance audit trails, and more. See the [Middleware catalog](#middleware-catalog) below.

The package also ships **first-class helpers** for RAG orchestration (`ragChain`), LLM-as-judge scoring (`llmJudge`), CI eval harnesses (`promptRegression`, `lintPrompt`), and offline batch workflows (`runBatch`, `waitForBatch`) — plus a `saptarishi-llm` CLI with subcommands for `chat`/`stream`/`embed`/`verify`/`init`/`mcp`/`chain-visualize`/`doctor`/`export-dashboard`/`lint-prompts` and more.

Complementary to [`@cap-js/ai`](https://github.com/cap-js/ai), which focuses on value-help recommendations and SAP AI Core integration. This plugin fills the more general "I need a CAP-idiomatic way to call LLMs, with a local development story and multiple provider options" gap.

## Middleware catalog

Every primitive below is a shipping middleware (or top-level helper) with dedicated per-section documentation, tests, TypeScript types, and a `config://<primitive>` MCP resource for live introspection. All are stable in the 2.x line — see [`MIGRATION.md`](./MIGRATION.md).

| Group | Primitives |
| --- | --- |
| **Cost** | `usageMetering` · `usageMeteringToCap` · `costBudget` · `costGuard` · `costForecast` · `quotaManager` (v2.23.0 — per-user USD quota with warnings) · `inMemoryQuotaStore` · `costOverrunPredictor` (v2.33.0 — calendar-window spend projection) · `startOfMonth` / `endOfMonth` (+day / quarter helpers) · `adaptiveMaxTokens` · `estimateCost` · `promptCacheStats` |
| **Resilience** | `retryOnRateLimit` · `circuitBreaker` · `bulkhead` · `adaptiveBulkhead` · `adaptiveRateLimit` · `clientSideRateLimit` (v2.26.0 — proactive N-per-window throttle) · `deadline` · `gracePeriod` (v2.28.0 — soft-deadline warnings + optional hard timeout) · `chatWithFallback` · `regionFailover` · `autoRetry` · `providerHealthProbe` · `autoContinue` · `idempotency` · `distributedLock` · `speculativeHedge` (v2.12.0 — staggered parallel hedges for tail latency) · `retryBudget` (v2.15.0 — SRE-style global retry cap) |
| **Security** | `guardrails` · `promptInjectionGuard` · `piiRedact` · `reversibleTokenization` (v2.13.0 — round-trip PII replacement) · `tokenizePII` / `detokenizePII` · `PII_PATTERNS` · `safetyClassifier` · `sensitiveDataAudit` · `requestSigning` (v2.21.0 — HMAC receipts + `verifyReceiptChain`) · `responseSigning` (v2.32.0 — HMAC responses + `verifyResponseSignature`) · `emptyResponseDetector` (v2.37.0 — catch empty/refusal responses with auto-retry) |
| **Observability** | `jsonLog` · `otel` · `otelSpans` · `promMetrics` · `prometheusHandler` · `traceCorrelation` · `healthHandler` · `replayBuffer` · `retryAfterPropagation` · `providerHealthAggregate` (v2.31.0 — unified provider health score) · `latencyHistogram` (v2.35.0 — per-dimension p50/p95/p99 with Prometheus export) |
| **Testing** | `chaosInjector` (v2.11.0 — deterministic seeded fault injection; test-only, refuses to construct without opt-in) |
| **Routing** | `modelRouter` · `tenantIsolate` · `costAwareRouter` (v2.10.0 — cheap-first with quality escalation) · `fairShareScheduler` (v2.14.0 — per-tenant WRR admission control) · `semanticRouter` (v2.16.0 — embedding-based route selection) · `providerLoadBalancer` (v2.17.0 — rotate across N credentials of same kind) · `multimodalRouter` (v2.25.0 — capability-aware routing by attachment type) |
| **Caching** | `responseCache` (exact + semantic) · `semanticCache` (pluggable vector store) · `cosineSimilarity` · `embeddingDedup` · `requestCoalescer` (in-flight dedup) · `fuzzyDedup` (v2.38.0 — near-duplicate detection via Jaccard-trigram / Levenshtein, no embedder required) · `inMemoryFuzzyStore` |
| **Contract / GitOps** | `structuredOutputValidator` · `structuredOutputRepair` (v2.9.0 — multi-strategy repair) · `jsonAutoFix` · `functionCallArbitrator` (v2.18.0 — tool-call allowlist + validation) · `normalizeToolShape` · `normalizeToolList` · `contentLengthGate` (v2.36.0 — pre-flight size validation) · `defaultTokenEstimator` · `schemas` (Invoice, PurchaseOrder, SupplierRisk, ContractSummary, ExpenseReport, EmailDraft) · `validateMiddlewareOrder` · `chainSnapshot` · `chainDiff` · `preflight` · `capabilities` (+ `PROVIDER_CAPABILITY_MATRIX`, `MODEL_CAPABILITY_OVERRIDES`) |
| **Prompts** | `PromptRegistry` · `builtInPrompts` · `gitPromptRegistry` (v2.1.0 — Git-backed prompt-as-code) · `promptVersionPin` (v2.30.0 — canary/rollback for prompt templates) · `PromptVersionRegistry` |
| **Long-context** | `compactHistory` · `sessionContextStore` (v2.19.0 — per-session history with prune / summarize) · `inMemorySessionStore` |
| **Streaming** | `wrapStreamCompletion` · `hasStreamCompletion` · `streamThrottle` · `streamAggregator` (v2.24.0 — coalesce per-token chunks for smoother UI) |
| **RAG + Eval** | `ragChain` · `llmJudge` / `judgeMany` · `promptRegression` / `loadFixtures` / `formatRegressionReport` · `lintPrompt` / `lintPrompts` / `formatLintReport` · `scoreResponse` · `consensusVoting` · `promptExperiment` (v2.20.0 — live A/B testing with 95% CI winner detection) · `responseRevision` (v2.29.0 — quality-driven re-ask loop) · `userFeedbackAggregator` (v2.34.0 — human thumbs/stars aggregation) |
| **Bulk workflows** | `runBatch` · `waitForBatch` · `batchAggregator` (v2.27.0 — window-based pooling of concurrent LLM calls) |
| **Multimodal helpers** | `imageFromFile` / `pdfFromUrl` / `audioFromBase64` / `uploadPdfFromUrl` / etc. |
| **Agent orchestration** | `runTools` · `streamTools` · `Agent` · `runAgents` · `streamAgents` · `autoToolChain` (v2.22.0 — cascading tool loop with cycle detection) |
| **Error taxonomy** | `LLMError` base + `errorRegistry` with 18 stable codes (`CIRCUIT_OPEN`, `BUDGET_EXCEEDED`, `PROMPT_INJECTION`, `SAFETY_CLASSIFIER_BLOCKED`, `ALL_REGIONS_FAILED`, ...) |

Every primitive is composable via `llm.use(mw)` Koa-style — outermost first. `resilience.bundle()` wires the full resilience stack (retry → breaker → bulkhead → deadline → probes → tuner) with one call. `chainSnapshot(llm)` extracts the live config for GitOps drift detection; `chainDiff(baseline, live)` diffs; `validateMiddlewareOrder(chain)` warns on suspicious orderings.

## Architecture

```
    Your CAP handler / OData action
          │
          │  cds.connect.to('llm')  →  { chat, stream, embed, batch }
          ↓
    ┌─────────────────────────────────────────────────────────────┐
    │  Middleware chain (llm.use(...), Koa-style, outer → inner)  │
    │                                                             │
    │   otelSpans / traceCorrelation / jsonLog                    │
    │     → promptCacheStats                                      │
    │       → modelRouter                                         │
    │         → embeddingDedup                                    │
    │           → safetyClassifier                                │
    │             → autoContinue                                  │
    │               → promptInjectionGuard                        │
    │                 → guardrails                                │
    │                   → piiRedact                               │
    │                     → sensitiveDataAudit                    │
    │                       → costGuard / costBudget              │
    │                         → adaptiveMaxTokens                 │
    │                           → idempotency / distributedLock   │
    │                             → circuitBreaker                │
    │                               → bulkhead                    │
    │                                 → retryOnRateLimit          │
    │                                   → responseCache           │
    │                                     → structuredOutputValidator
    │                                       → replayBuffer        │
    │                                         → (provider)        │
    └─────────────────────────────────────────────────────────────┘
                 │
                 ▼
    ┌─────────────────────────────────────────────────────────────┐
    │  LLMService (base class): retries · structured-output       │
    │  parsing · unified chunk shape · batch API dispatch         │
    └───────────────────┬─────────────────────────────────────────┘
                        │
                        ▼
    ┌─── AnthropicLLM ────┬─── AzureOpenAILLM ──┬─── GeminiLLM ─┐
    │    OpenAICompatible │    GroqLLM          │    BedrockLLM │
    │    FireworksLLM     │    DeepSeekLLM      │    MistralLLM │
    │    OllamaLLM        │    GenAIHubLLM      │               │
    └─────────────────────┴─────────────────────┴───────────────┘
```

The middleware sandwich is **fully composable** — every layer is optional, layer order is validated by `validateMiddlewareOrder`, and the full snapshot is exportable for GitOps drift detection via `chainSnapshot`. Each layer exposes an `asMcpResource()` handler for live observability.

- **No CDS entities or served OData surface** — this is a client library, not an OData service. `cds.connect.to('llm')` returns the provider instance directly.
- **Provider selection at connect time** via `cds.requires.llm.kind` — profile-aware, so `[development]`, `[production]`, `[genai-hub]`, etc. can each point at a different backend.
- **Provider inheritance:** `GroqLLMService` and `GenAIHubLLMService` both extend `OpenAICompatibleLLMService` (they speak the OpenAI `/chat/completions` shape); the latter adds OAuth + resource-group headers on top.

## Install

```bash
npm install @saptarishi/cds-plugin-llm
```

Optional peer dep for the Anthropic path: `@anthropic-ai/sdk` (installed automatically as a dependency).

**TypeScript:** full type definitions ship in the package (`lib/index.d.ts`). No `@types/*` package needed.

**CLI:** a `saptarishi-llm` executable ships with the package. Use it via `npx` without installing globally, or install once with `npm i -g @saptarishi/cds-plugin-llm`. See the [CLI section](#cli-new-in-v150).

## Configure

Add to your CAP app's `package.json` under `cds.requires`:

```json
{
  "cds": {
    "requires": {
      "llm": {
        "[development]": { "kind": "llm-groq",       "modelId": "llama-3.3-70b-versatile" },
        "[ollama]":      { "kind": "llm-ollama",     "modelId": "qwen2.5:14b"             },
        "[production]":  { "kind": "llm-genai-hub", "credentials": { "deploymentId": "..." } }
      }
    }
  }
}
```

Set the appropriate env var (see `.env.example`):
- `ANTHROPIC_API_KEY` for `llm-anthropic`
- `OLLAMA_BASE_URL` for `llm-ollama` (defaults to `http://localhost:11434`)
- `GROQ_API_KEY` for `llm-groq`
- `OPENAI_API_KEY` + `OPENAI_BASE_URL` for `llm-openai-compatible`

## Providers

| Kind | Backend | Cost to test | Status |
|---|---|---|---|
| `llm-anthropic` | Claude via Anthropic API | Pennies per call | Working |
| `llm-ollama` | Local Ollama daemon | Free | Working |
| `llm-groq` | Groq's hosted Llama/Mixtral/Qwen (sub-second inference) | Generous free tier | Working |
| `llm-openai-compatible` | Any endpoint speaking OpenAI's `/chat/completions` (OpenAI, Together, Fireworks, DeepSeek direct, LM Studio, LocalAI...) | Varies | Working |
| `llm-azure-openai` | Azure OpenAI Service (per-deployment URL + `api-key` header) | Paid via Azure subscription | Working (mock-verified end-to-end) |
| `llm-gemini` (new in 1.19.0) | Google Gemini via Google AI Studio (chat, streaming, tools, vision, embeddings) | Free tier + paid per token | Working (mock-verified end-to-end) |
| `llm-bedrock` (new in 1.19.0) | AWS Bedrock Converse API (chat, streaming, tools, vision) + Titan/Cohere embed. Uses `@aws-sdk/client-bedrock-runtime` (optional peer). | Paid via AWS Bedrock | Working (mock-verified end-to-end) |
| `llm-fireworks` (new in 1.23.0) | Fireworks AI — hosted OSS models (Llama, Qwen, Mixtral, DeepSeek) behind an OpenAI-compatible endpoint | Pay-per-token via Fireworks | Working (mock-verified end-to-end) |
| `llm-deepseek` (new in 1.23.0) | DeepSeek — direct API access to DeepSeek-V3 (chat) and DeepSeek-R1 (reasoning) | Very cheap per-token | Working (mock-verified end-to-end) |
| `llm-mistral` (new in 1.23.0) | Mistral AI — Mistral Large, Codestral, and the open-weights family | Pay-per-token via Mistral | Working (mock-verified end-to-end) |
| `llm-genai-hub` | SAP AI Core / Generative AI Hub | Paid (extended plan) | Spec-compliant · mock-verified end-to-end · live verify open (community help welcome) |

## Stability

- **Semantic Versioning.** `1.x` will preserve the public API contract documented in `lib/index.d.ts`. Breaking changes require a major version bump.
- **What's covered:** all exported symbols in `lib/index.js` — `LLMService` and 5 provider subclasses, `chat` / `stream` / `embed` shapes, `ContentBlock` union (text / image / document), message shapes, tool call shapes, stream chunk shapes, image / PDF helpers, and provider option fields (`kind`, `modelId`, `credentials`, `retries`, `responseCache`).
- **What's not covered:** provider-native `raw` response objects (shape follows the upstream API), CDS kind config field names (which follow CAP conventions), and behavior of individual model IDs (upstream provider concern).
- **Deprecation policy:** any deprecated field or method will be marked in the JSDoc + CHANGELOG for at least one minor release before removal, and only removed in a subsequent major version.
- **Full history** in [`CHANGELOG.md`](./CHANGELOG.md).

## Use

Standard CAP idiom — `cds.connect.to()`:

```js
const cds = require('@sap/cds');

module.exports = class ProcurementService extends cds.ApplicationService {
  async init() {
    const llm = await cds.connect.to('llm');

    this.on('summarizePO', async (req) => {
      const { poId } = req.data;
      const po = await SELECT.one.from('PurchaseOrders').where({ ID: poId });

      const { text } = await llm.chat({
        system: 'You summarize purchase orders for approvers in 2 sentences.',
        messages: [{ role: 'user', content: JSON.stringify(po) }],
        cache: true,  // Anthropic-only: caches the system prompt
      });

      return text;
    });

    return super.init();
  }
};
```

## Structured outputs (new in v0.3.0)

Pass a JSON schema via `format` and get a parsed `.data` field back:

```js
const { data, usage } = await llm.chat({
  system: 'Assess supplier invoice risk for AP triage.',
  messages: [{ role: 'user', content: invoiceJson }],
  format: {
    type: 'object',
    properties: {
      risk: { type: 'string', enum: ['low', 'medium', 'high'] },
      rationale: { type: 'string' },
    },
    required: ['risk', 'rationale'],
    additionalProperties: false,
  },
});

console.log(data.risk);       // 'high'
console.log(data.rationale);  // 'Amount over 100k EUR without matched PO...'
```

Under the hood:
- **Anthropic**: uses `output_config.format` (native JSON schema)
- **OpenAI-compatible / Groq**: uses `response_format: { type: 'json_object' }` and prepends the schema to the system prompt for broadest model coverage
- **Ollama**: uses native `format` field (schema-strict on recent Ollama versions)
- **Base class** post-parses `.text` into `.data` uniformly; falls back to first-`{...}`-block extraction if the model wrapped the JSON in prose

## Tool use / function calling (new in v0.3.0)

Pass a unified tool schema; get normalized `toolCalls` back:

```js
const turn1 = await llm.chat({
  system: 'Help procurement approvers. Use tools to fetch data.',
  messages: [{ role: 'user', content: 'Fetch PO 4500000123' }],
  tools: [{
    name: 'get_purchase_order',
    description: 'Fetch a purchase order by its 10-digit ID',
    input_schema: {
      type: 'object',
      properties: { purchaseOrderId: { type: 'string' } },
      required: ['purchaseOrderId'],
    },
  }],
});

if (turn1.toolCalls?.length) {
  const call = turn1.toolCalls[0];  // { id, name, input }
  const result = await fetchPO(call.input.purchaseOrderId);  // your app logic

  // Feed the result back for turn 2
  const turn2 = await llm.chat({
    system: '...',
    messages: [
      { role: 'user',      content: 'Fetch PO 4500000123' },
      { role: 'assistant', toolCalls: turn1.toolCalls },
      { role: 'tool',      tool_call_id: call.id, content: JSON.stringify(result) },
    ],
    tools: [...],
  });
  console.log(turn2.text);  // model's final answer
}
```

Works across providers with matching `{ id, name, input }` shape. Individual model quality varies for multi-tool scenarios — llama-3.3-70b on Groq is solid for single-tool cases; Claude and qwen2.5 are more reliable for chained tool use.

### Tool runner — automatic multi-turn loop (new in v1.1.0)

For agent-style code that would otherwise write the "call → execute → feed back → repeat" loop by hand, `runTools()` wraps the pattern:

```js
const { runTools } = require('@saptarishi/cds-plugin-llm');

const result = await runTools({
  llm,
  system: 'You help procurement approvers.',
  messages: [{ role: 'user', content: 'Fetch PO 4500000123 and summarize it.' }],
  tools: [{
    name: 'get_purchase_order',
    description: 'Fetch a PO by 10-digit ID',
    input_schema: {
      type: 'object',
      properties: { purchaseOrderId: { type: 'string' } },
      required: ['purchaseOrderId'],
    },
    run: async ({ purchaseOrderId }) =>
      await SELECT.one.from('PurchaseOrders').where({ ID: purchaseOrderId }),
  }],
  maxSteps: 10,
});

result.text        // final assistant answer
result.steps       // number of chat() calls made
result.toolCalls   // [{ id, name, input, result, isError }, ...] — every call executed
result.usage       // aggregated tokens across all turns
```

What it handles for you:
- Executes every tool call in every turn (multiple in one turn all run)
- Appends assistant + `tool` messages correctly (matches the format both Anthropic and OpenAI-compat expect)
- Catches tool exceptions and surfaces them as `tool_result` with `is_error: true` so the model can recover
- Rejects with a clear message when an unknown tool name is called
- `maxSteps` safety cap (default 10) — throws if the model loops forever
- Optional `onStep({ step, response })` callback to observe each turn

## Streaming (new in v0.6.0)

Get tokens as they arrive from the model instead of waiting for the full response:

```js
for await (const chunk of llm.stream({
  system: 'You are a poet.',
  messages: [{ role: 'user', content: 'Write a haiku about SAP procurement.' }],
})) {
  if (chunk.type === 'text_delta') {
    process.stdout.write(chunk.text);
  }
  if (chunk.type === 'done') {
    console.log(`\n[${chunk.usage.output_tokens} tokens, stopReason: ${chunk.stopReason}]`);
  }
}
```

Chunk types:

| type | payload | when |
|---|---|---|
| `text_delta` | `{ text }` — incremental piece of text | fires per token/token-group as the model generates |
| `done` | `{ text, usage, stopReason, model }` — accumulated text + final metadata | once at the end |

Wire-shape parsing per provider:
- **Anthropic**: uses the SDK's `messages.stream()` — SSE under the hood, events adapted to unified chunks
- **OpenAI-compatible / Groq / GenAI Hub**: parses SSE (`data: {json}\n\n`) from the `/chat/completions` streaming endpoint, adds `stream_options: {include_usage: true}` so `usage` populates on the `done` chunk
- **Ollama**: parses NDJSON from `/api/chat`, emits `done` when the stream's final message carries `done:true`

Retries are **not** applied to streams (partial-response semantics are unclear). If a stream fails mid-way, the caller sees the error thrown from the generator.

Try the demo:

```sh
node scripts/stream-demo.js "Explain streaming LLM responses in 3 sentences."
```

## Embeddings (expanded in v0.7.0)

```js
const { embeddings } = await llm.embed({
  input: ['first document', 'second document', 'third'],
  model: 'text-embedding-3-small',  // optional; falls back to configured modelId
});
// embeddings is number[][] — one vector per input string
```

Supported providers:
- **Ollama** — `mxbai-embed-large`, `nomic-embed-text`, `all-minilm`, any embedding model you've pulled
- **OpenAI-compatible** (including Groq, Together AI, DeepSeek, LM Studio) — `text-embedding-3-small`, `text-embedding-3-large`, `text-embedding-ada-002`, provider-specific models
- **Anthropic**: not supported (no first-party embeddings)
- **GenAI Hub**: needs a separate embedding-model deployment; not yet plumbed (planned for 0.9)

Single string or array of strings both work. Returns `{ embeddings: number[][], model: string }` — the outer array always matches the input length.

## Vision / multimodal input (new in v0.5.0)

Pass images inline as content blocks. Works across all providers with vision-capable models (Claude 3.5+, GPT-4o, Groq's `llama-3.2-*-vision`, Ollama's `llava` / `moondream` / `llama3.2-vision`).

```js
const { imageFromFile, imageFromUrl, imageFromBase64 } = require('@saptarishi/cds-plugin-llm');

// Load from disk
const image = await imageFromFile('/tmp/scanned-invoice.png');

// Or from a URL (Anthropic + OpenAI-compat; Ollama needs base64)
const image = imageFromUrl('https://example.com/invoice.png');

// Or from base64 data you already have
const image = imageFromBase64(base64Data, 'image/png');

const { data } = await llm.chat({
  model: 'gpt-4o',  // or claude-opus-4-7, llama-3.2-11b-vision-preview, llava, ...
  system: 'Extract structured data from scanned invoices.',
  messages: [{
    role: 'user',
    content: [
      image,
      { type: 'text', text: 'Return the vendor, invoice number, and line items.' },
    ],
  }],
  format: {
    type: 'object',
    properties: {
      vendor: { type: 'string' },
      invoiceNumber: { type: 'string' },
      lineItems: {
        type: 'array',
        items: {
          type: 'object',
          properties: {
            description: { type: 'string' },
            quantity: { type: 'number' },
            unitPrice: { type: 'number' },
          },
        },
      },
    },
  },
});
```

Wire-shape translation is provider-aware:
- **Anthropic**: native content blocks (source can be `url` or `base64`)
- **OpenAI-compatible / Groq**: `image_url` blocks with data URLs for base64
- **Ollama**: text goes in `content`, images extracted to `images: [base64, ...]` (Ollama does not accept URLs — use `imageFromFile()` or `imageFromBase64()`)

## PDF documents (v0.8.0 · expanded in v0.9.0)

Pass PDF documents inline. Full native support on Anthropic (Claude 3.5+ parses text + visuals in one pass). Since **v0.9.0** OpenAI-compat providers accept base64 PDFs too via the `file` content-block shape — works on GPT-4o and newer OpenAI models. Groq and other OpenAI-compat providers that don't accept files will 400 upstream.

```js
const { pdfFromFile, pdfFromUrl, pdfFromBase64 } = require('@saptarishi/cds-plugin-llm');

const pdf = await pdfFromFile('/tmp/scanned-invoice.pdf');
// or: const pdf = pdfFromUrl('https://example.com/invoice.pdf');
// or: const pdf = pdfFromBase64(base64Data);

const { data } = await llm.chat({
  model: 'claude-opus-4-7',
  system: 'Extract structured data from scanned invoices.',
  messages: [{
    role: 'user',
    content: [
      pdf,
      { type: 'text', text: 'Return vendor, invoice number, line items.' },
    ],
  }],
  format: { /* JSON schema */ },
});
```

Provider notes:
- **Anthropic** — native, both base64 and URL sources
- **OpenAI-compat (GPT-4o+)** — base64 works out of the box. **URL PDFs (new in v1.14.0)** via the Files API: call `uploadPdfFromUrl(url, {apiKey})` first — it fetches, uploads to `/v1/files`, and returns a document block with `source.type='file_id'` that the provider translates to `{type:'file', file:{file_id}}`.
- **Groq / other OpenAI-compat that don't accept files** — base64 will 400 upstream; `/v1/files` isn't exposed so `uploadPdfFromUrl` will also 404 there
- **Ollama** — no PDF support; render pages to images via `pdftoppm` (poppler) and pass to a vision model like `llava` or `llama3.2-vision`

### URL PDFs on OpenAI (v1.14.0+)

```js
const { uploadPdfFromUrl } = require('@saptarishi/cds-plugin-llm');

const doc = await uploadPdfFromUrl('https://example.com/contract.pdf', {
  apiKey: process.env.OPENAI_API_KEY,
  // baseUrl:  'https://api.openai.com/v1'  (default)
  // purpose:  'user_data'                  (default)
  // filename: <inferred from URL basename, .pdf appended if missing>
});

await openai.chat({
  messages: [{ role: 'user', content: [doc, { type: 'text', text: 'Summarize' }] }],
});
```

The returned document block has `source: { type: 'file_id', file_id: 'file-xxx' }`. Reuse the block across multiple chat calls to avoid re-uploading. OpenAI retains uploaded files by account — bring your own retention policy.

## SAP Generative AI Hub setup

The `llm-genai-hub` kind targets a **deployment** in your BTP AI Core instance. Prerequisites:

1. **Provision AI Core** — BTP Cockpit → Service Marketplace → *AI Core* → **extended** plan (free plan does not include Generative AI Hub).
2. **Create a resource group** (or use `default`).
3. **Deploy a model** via SAP AI Launchpad, `ai-api-cli`, or the SDK — e.g. `gpt-4o`, `mistral-large-instruct`, `claude-3-5-sonnet`. Note the deployment ID.
4. **Configure the plugin** — three ways depending on where your CAP app runs.

### On BTP Cloud Foundry (recommended)

Bind the AI Core service instance to your CAP app:

```sh
cf bind-service <your-app> <ai-core-instance>
cf restage <your-app>
```

Then set only the deployment ID (credentials auto-discovered from `VCAP_SERVICES`):

```sh
cf set-env <your-app> AICORE_DEPLOYMENT_ID <deployment-id>
cf set-env <your-app> AICORE_RESOURCE_GROUP default   # optional; defaults to 'default'
```

In `package.json`:

```json
{
  "cds": { "requires": { "llm": {
    "[production]": { "kind": "llm-genai-hub", "modelId": "gpt-4o" }
  }}}
}
```

### On Kyma

Attach the service binding manifest, then set the same env vars via a ConfigMap or Secret. The `VCAP_SERVICES` layout is preserved by the SBO (Service Binding Operator).

### Local dev pointing at a BTP-hosted AI Core

Extract the service key JSON from BTP Cockpit (Service Instance → Service Keys → View). Put values in `.env`:

```
AICORE_API_URL=https://api.ai.prod.eu-central-1.aws.ml.hana.ondemand.com
AICORE_AUTH_URL=https://<subaccount>.authentication.<region>.hana.ondemand.com
AICORE_CLIENT_ID=sb-...
AICORE_CLIENT_SECRET=...
AICORE_DEPLOYMENT_ID=abc123                # chat model deployment
AICORE_EMBEDDING_DEPLOYMENT_ID=def456      # optional; enables llm.embed()
AICORE_MODEL=gpt-4o
```

Or pass explicitly in `package.json`:

```json
{
  "cds": { "requires": { "llm": {
    "[genai-hub]": {
      "kind": "llm-genai-hub",
      "modelId": "gpt-4o",
      "credentials": {
        "aiCoreUrl":     "https://api.ai.prod.eu-central-1.aws.ml.hana.ondemand.com",
        "tokenUrl":      "https://<subaccount>.authentication.<region>.hana.ondemand.com",
        "clientId":      "sb-...",
        "clientSecret":  "...",
        "deploymentId":  "abc123",
        "resourceGroup": "default"
      }
    }
  }}}
}
```

### What it handles for you

- OAuth2 client-credentials flow against XSUAA
- Token caching + refresh (60s before expiry)
- `AI-Resource-Group` header
- Deployment-based inference endpoint construction
- `VCAP_SERVICES.aicore` auto-discovery when the service is bound

### Known limitations (v0.4.0)

- **OpenAI-shape only.** Deployments that expose the OpenAI `/chat/completions` shape (GPT, Mistral, Llama, Gemini, and Anthropic-via-shim) work. Native Anthropic-shape deployments (Claude via `/invoke`) are not yet supported — use the `llm-anthropic` kind directly for Claude.
- **Not yet live-verified.** Built to the SAP-documented API contract and unit-tested against mocks. Live verification against an AI Core `extended` deployment is the next contribution wanted.

## Automatic retries (new in v0.3.0)

Every `chat()` and `embed()` call is wrapped with exponential-backoff retry on 429 / 5xx responses. Honors `Retry-After` headers. Configurable per-call or globally:

```js
// Per-call override
await llm.chat({ messages: [...], retries: { max: 5, baseMs: 1000, maxMs: 30000 } });

// Or via cds.requires.llm config:
{ "cds": { "requires": { "llm": {
  "kind": "llm-groq",
  "retries": { "max": 5 }
}}}}
```

## Middleware / interceptors (new in v1.2.0)

Register hooks around every `chat` / `stream` / `embed` call. Koa-style compose — outermost first, `next()` returns the next middleware's result (or the provider's response). Middleware may inspect or transform the request AND the response, share state via `ctx.meta`, or short-circuit by returning without calling `next()`.

```js
const llm = await cds.connect.to('llm');

// 1. Logging + duration
llm.use(async (ctx, next) => {
  const start = Date.now();
  const res = await next();
  console.log(`[${ctx.method}] ${Date.now() - start}ms`);
  return res;
});

// 2. Cost tracking (aggregate tokens across every call)
const totals = { in: 0, out: 0 };
llm.use(async (ctx, next) => {
  const res = await next();
  totals.in  += res?.usage?.input_tokens  ?? 0;
  totals.out += res?.usage?.output_tokens ?? 0;
  return res;
});

// 3. Auto-injected system prompt suffix
llm.use(async (ctx, next) => {
  if (ctx.method === 'chat' && ctx.request.system) {
    ctx.request.system += '\n\nBe concise. If uncertain, say so.';
  }
  return next();
});

// 4. Streams: wrap the iterator to observe each chunk
llm.use(async (ctx, next) => {
  if (ctx.method !== 'stream') return next();
  const inner = await next();
  return (async function* () {
    for await (const chunk of inner) {
      if (chunk.type === 'text_delta') myLiveUI.append(chunk.text);
      yield chunk;
    }
  })();
});
```

Context object:
- `ctx.method` — `'chat'` | `'stream'` | `'embed'`
- `ctx.request` — mutable request options (modify before `next()` to affect the provider call)
- `ctx.meta` — scratchpad for cross-middleware state (e.g. timing marks, request IDs)

Notes:
- Middleware runs **around** the response cache, retries, and format-parsing — those are internal concerns your middleware can observe. Cache hits arrive at your middleware with `cached: true` set.
- Calling `next()` more than once from the same middleware throws.
- Errors propagate up the chain.
- For streams, `next()` returns an async iterable. To observe/transform chunks, wrap it into a new async generator.

### Built-in middleware (new in v1.3.0)

Two production-oriented middlewares ship in the box: rate limiting and OpenTelemetry tracing.

#### `rateLimit` — token-bucket limiter

```js
const { rateLimit } = require('@saptarishi/cds-plugin-llm');

const llm = await cds.connect.to('llm');

// Global: 60 requests burst, refill at 1/s
llm.use(rateLimit({ capacity: 60, refillPerSecond: 1 }));

// Per-user: keyed off ctx.meta.user (populate this from an earlier middleware
// that inspects your CAP request)
llm.use(rateLimit({
  capacity: 10,
  refillPerSecond: 0.2,
  keyFn: (ctx) => ctx.meta.user ?? 'anon',
  mode: 'wait',   // 'throw' (default) or 'wait' — pause instead of erroring
}));
```

When `mode: 'throw'` and the bucket is empty, the middleware throws an `Error` with `code: 'RATE_LIMITED'` and `retryAfterMs` so you can surface a proper 429 to your caller. Buckets are in-process — for multi-instance CF apps that need a shared counter, back with Redis via your own middleware.

#### `otel` — OpenTelemetry spans

```js
const { trace } = require('@opentelemetry/api');
const { otel } = require('@saptarishi/cds-plugin-llm');

llm.use(otel({
  tracer: trace.getTracer('cap-app'),
  systemAttribute: 'anthropic',   // sets gen_ai.system on every span
}));
```

Emits one span per `chat` / `stream` / `embed` call. Attributes follow the emerging GenAI semantic conventions where possible: `gen_ai.system`, `gen_ai.operation.name`, `gen_ai.request.model`, `gen_ai.response.model`, `gen_ai.usage.input_tokens` / `output_tokens`, `gen_ai.response.stop_reason`. Plus a few plugin-specific ones: `llm.cached`, `llm.tool_calls.count`, `llm.stream.chunks`, `llm.embed.count`. Duck-typed against `@opentelemetry/api` — no hard dependency, works with any object exposing `startSpan()`.

Stream spans stay open through the whole iterator (span ends on the `done` chunk, on early break, or on error — never leaks).

#### `redisRateLimit` — shared bucket across CF instances (new in v1.4.0)

The in-process `rateLimit` is fine for single-instance apps. For multi-instance CF deployments where a shared quota must be enforced globally, back the bucket with Redis.

```js
const Redis = require('ioredis');
const { redisRateLimit } = require('@saptarishi/cds-plugin-llm');

llm.use(redisRateLimit({
  redis: new Redis(process.env.REDIS_URL),
  capacity: 60,
  refillPerSecond: 1,
  keyFn: (ctx) => ctx.meta.user ?? 'anon',
  keyPrefix: 'ratelimit:llm:',   // default 'saptarishi:llm:rl:'
  mode: 'throw',                  // 'throw' | 'wait'
}));
```

Uses an atomic Lua `EVAL` so two instances checking the bucket at the same time cannot both succeed when only one token is left. Duck-typed client — any object with an `eval(script, numKeys, ...args)` promise API satisfies (works with `ioredis` and `node-redis` v4+ out of the box). On BTP, bind a Redis service to your CF app and pull the URL from `VCAP_SERVICES`.

## CLI (new in v1.5.0)

A `saptarishi-llm` executable ships with the package. Handy for provider health checks in CI, quick prompt experiments from the shell, pipelining embeddings into a downstream tool, or **scaffolding a fresh CAP project** pre-wired to the plugin (v1.6.0).

```bash
npx @saptarishi/cds-plugin-llm --help

# or install globally
npm install -g @saptarishi/cds-plugin-llm
saptarishi-llm --help
```

### Commands

```bash
saptarishi-llm chat -p "explain SAP CAP in one sentence"
saptarishi-llm stream -p "write a haiku about procurement"
saptarishi-llm embed -p "purchase order for steel coils" --json
saptarishi-llm verify --provider anthropic
saptarishi-llm providers
```

### Provider selection

`--provider <kind>` or `SAPTARISHI_LLM_PROVIDER` env var. Same five kinds as the CAP plugin: `anthropic`, `ollama`, `groq`, `openai-compatible`, `genai-hub`.

Credentials come from env vars (never CLI flags — avoids leaking secrets into shell history):

```bash
ANTHROPIC_API_KEY=sk-ant-... saptarishi-llm chat -p "hello"
OLLAMA_URL=http://192.168.5.13:11434 saptarishi-llm chat --provider ollama -p "hello"
GROQ_API_KEY=gsk-... saptarishi-llm verify --provider groq
```

### Input sources

Prompt can come from `--prompt` / `-p`, `--file` / `-f`, stdin, or a positional arg. Multiple sources concatenate with a blank line between them.

```bash
echo "summarize this" | saptarishi-llm chat -f contract.pdf.txt
saptarishi-llm embed -p "one\n---\ntwo\n---\nthree"      # 3 vectors from 1 call
```

### CI health checks

`verify` connects, runs a tiny probe, reports latency, and exits `0` on success / `1` on unexpected reply / `1` on error. Drop it in a nightly workflow to catch expired credentials or endpoint outages before your CAP app does.

```bash
saptarishi-llm verify --provider genai-hub --json
```

### Expose as an MCP server (new in v1.7.0)

`saptarishi-llm mcp` runs a Model Context Protocol server over stdio that exposes the configured provider as tools any MCP client can call. Register it in Claude Desktop, Cursor, Zed, or any other MCP-capable client and those clients gain a `chat` / `embed` / `verify` / `list_providers` tool backed by **your** provider config — with all its middleware, caching, rate limits, and OTel tracing.

The point: one MCP server = "the sanctioned way to call an LLM at MyCompany". Centralized credentials, centralized policy, developer productivity everywhere.

Claude Desktop (`~/Library/Application Support/Claude/claude_desktop_config.json` on macOS):

```jsonc
{
  "mcpServers": {
    "saptarishi-llm": {
      "command": "npx",
      "args": ["-y", "@saptarishi/cds-plugin-llm", "mcp"],
      "env": {
        "SAPTARISHI_LLM_PROVIDER": "groq",
        "GROQ_API_KEY": "gsk-..."
      }
    }
  }
}
```

Tools exposed:

| Tool | Purpose |
|------|---------|
| `chat` | Send a prompt, return text. `{ prompt, system?, maxTokens? }` |
| `embed` | Embed input(s) into vectors. `{ input: string \| string[] }` |
| `verify` | Tiny probe against the provider. Returns `{ ok, latencyMs, model, text }`. |
| `list_providers` | Enumerate every supported provider kind with default models. |

Uses a hand-rolled MCP implementation (2024-11-05 spec) over stdio JSON-RPC 2.0 — **zero new dependencies**. Full protocol coverage: `initialize`, `tools/list`, `tools/call`, `resources/list`, `resources/read`, `resources/subscribe`, `resources/unsubscribe`, `prompts/list`, `prompts/get`, `ping`, notifications. Tool errors surface as `result.isError: true` per spec so the model can recover, not as JSON-RPC errors.

**Resources exposed** (readable via `resources/read`):

| URI | Description |
|-----|-------------|
| `config://active-provider` | Current provider kind + model + middleware count. |
| `config://supported-providers` | Every provider kind + default model. |

**Prompts registered** (invokable via `prompts/get`) — see [Prompt-template registry](#prompt-template-registry-new-in-v180) below for the full built-in list.

**Resource templates** (new in v1.9.0; parametrized URIs discoverable via `resources/templates/list`):

| URI template | What clients can read |
|---|---|
| `provider://{kind}` | Default model for a specific provider kind (`provider://groq` → `{ kind: 'groq', defaultModel: 'llama-3.3-70b-versatile' }`). |
| `prompt://{name}` | Metadata (arguments, description) for a registered prompt template. To render, use `prompts/get`. |

### Scaffold a fresh CAP project (new in v1.6.0)

`init` creates a fully-wired CAP app in seconds — no manual `package.json` editing, no CDS boilerplate:

```bash
npx @saptarishi/cds-plugin-llm init joule-demo --provider groq
cd joule-demo
cp .env.example .env               # then fill in real credentials
npm install
cds watch
```

Then:

```bash
curl 'http://localhost:4004/ai/chat(prompt='"'"'hello'"'"')'
```

Generated:

```
joule-demo/
├── package.json          # cds.requires.llm pointing at chosen provider
├── srv/
│   ├── ai-service.cds    # service AIService { chat(prompt), summarize(text) }
│   └── ai-service.js     # handlers using cds.connect.to('llm')
│                         # + SSE streaming endpoint at POST /stream/chat
├── .env.example          # provider-specific env vars
├── .gitignore            # excludes .env, node_modules/, gen/
└── README.md             # how to run
```

**Streaming out of the box (v1.9.0+):** the generated `srv/ai-service.js` registers `POST /stream/chat` inline, streaming tokens as they arrive:

```bash
curl -N -X POST http://localhost:4004/stream/chat \
  -H 'content-type: application/json' \
  -d '{"prompt":"write a haiku about SAP CAP"}'
```

Flags:
- `--provider <kind>` — `anthropic` (default) | `ollama` | `groq` | `openai-compatible` | `genai-hub`
- `--model <id>` — override the default model for the chosen provider
- `--force` — overwrite a non-empty target directory
- `--dry-run` — print the file list without writing anything

## Prompt-template registry (new in v1.8.0)

Register named prompt templates once, invoke them by name from any CAP handler, and automatically expose them over MCP so external clients (Claude Desktop, Cursor, Zed) can invoke them too. Same registry, three surfaces.

```js
const { PromptRegistry, builtInPrompts } = require('@saptarishi/cds-plugin-llm');

const prompts = new PromptRegistry()
  .registerAll(builtInPrompts())                    // 5 built-ins
  .register({                                       // your own
    name: 'invoice_dispute_response',
    description: 'Draft a courteous response to an invoice dispute.',
    arguments: [
      { name: 'dispute', required: true, description: 'The dispute text' },
      { name: 'tone',    required: false, description: 'formal | friendly' },
    ],
    render: ({ dispute, tone = 'formal' }) => ({
      system: `You are AP support. Reply in a ${tone} tone. Never admit liability.`,
      messages: [{ role: 'user', content: dispute }],
    }),
  });

// From a CAP handler:
const req = prompts.render('invoice_dispute_response', { dispute });
const res = await llm.chat({ ...req, maxTokens: 512 });
```

Built-ins (`builtInPrompts()`):

| Name | Purpose |
|------|---------|
| `summarize` | Condense text to N sentences (`text`, `sentences?`). |
| `extract_json` | Extract structured JSON against a schema (`text`, `schema`). |
| `classify` | Assign one label from a set (`text`, `labels`). |
| `translate` | Translate to a target language (`text`, `targetLanguage`). |
| `procurement_risk_scorer` | SAP-flavored risk analyst prompt (`text`). |

Templates auto-appear as MCP prompts when you run `saptarishi-llm mcp`. Claude Desktop shows them as slash-commands the user can pick.

### Load templates from a folder (new in v1.9.0)

Instead of registering templates inline, drop `*.mjs` or `*.js` files into a directory and load them at boot:

```
prompts/
├── invoice_dispute.mjs        # export default { name, render, ... }
├── kpi_extractor.mjs          # export default { ... }
└── shared.mjs                 # export const foo = ...; export const bar = ...
```

```js
await registry.loadFromDir('./prompts');
```

Or expose the whole folder via MCP without writing any code:

```bash
saptarishi-llm mcp --prompts-dir ./prompts
# or: SAPTARISHI_LLM_PROMPTS_DIR=./prompts saptarishi-llm mcp
```

Three export conventions handled in one scan: `export default <template>`, `export default [<t1>, <t2>]`, or named exports.

Add `--watch-prompts` and the server hot-reloads templates when files change — iterate without restart (new in v1.10.0):

```bash
saptarishi-llm mcp --prompts-dir ./prompts --watch-prompts
```

### Git-backed prompt registry (new in v2.1.0)

`gitPromptRegistry` extends `PromptRegistry` with a loader that pulls templates from a Git repository, caches them locally, and optionally polls for updates. Prompt changes go through PR review separately from code deploys — a real prompt-as-code workflow.

```js
const { gitPromptRegistry } = require('@saptarishi/cds-plugin-llm');

const registry = await gitPromptRegistry({
  url:     'https://github.com/your-org/prompts.git',
  branch:  'main',
  subdir:  'templates',      // optional — load only a subfolder
  pollMs:  60_000,           // poll every 60s for new commits
  onUpdate: (info) => cds.log('llm:prompts').info('reloaded', info),
});

// Same API as PromptRegistry:
const req = registry.render('invoice_dispute_response', { dispute });
```

- Exposes `config://git-prompt-registry` (URL, branch, current SHA, refresh timestamp, counters) — live-visible in your MCP client.
- Pin to a specific `ref` (SHA / tag) for immutable prompts in prod; use `branch` for rolling updates in staging.
- Works with any prompt directory the shipped `loadFromDir` scanner accepts.

### HTTP+SSE transport (new in v1.10.0)

By default `saptarishi-llm mcp` speaks stdio (for Claude Desktop / Cursor / Zed as a local subprocess). Add `--http` and it runs as a network service instead — the same MCP server, addressable over HTTP. Deploy to CF, put behind an auth proxy, share with a team.

```bash
saptarishi-llm mcp --http --port 3333 --host 0.0.0.0
# → MCP HTTP+SSE listening on http://0.0.0.0:3333/sse
# → GET /health for liveness/session count
```

Wire protocol (MCP 2024-11-05 SSE spec):

| Method | Path | Purpose |
|---|---|---|
| `GET` | `/sse` | Client opens SSE stream. Server sends `event: endpoint\ndata: /messages?sessionId=<uuid>`. Server replies to that session flow back on this stream. |
| `POST` | `/messages?sessionId=<uuid>` | Client sends a single JSON-RPC message. Server acknowledges 202; reply arrives on the SSE stream. |
| `GET` | `/health` | `{ server, version, transport, sessions }` — plug into monitoring. |

Multi-session — N concurrent clients each get their own session and stream. Graceful shutdown on SIGINT/SIGTERM.

#### Bearer-token auth (new in v1.11.0)

Deploying `--http` to anything beyond `127.0.0.1` should require a credential. Set `--auth-token` (or `SAPTARISHI_LLM_MCP_TOKEN` env) and every `/sse` and `/messages` request must send `Authorization: Bearer <token>`:

```bash
saptarishi-llm mcp --http --host 0.0.0.0 --port 3333 \
  --auth-token "$(openssl rand -hex 32)"
```

Behavior:
- Missing / wrong token → `401 Unauthorized` with `WWW-Authenticate: Bearer realm="mcp"`.
- `/health` stays **public** so load balancers can probe without credentials. Its response now includes `authRequired: <bool>` so clients know whether they need a token.
- Constant-time comparison on the byte-by-byte match to avoid trivial timing side-channels.
- Binding to a non-loopback host without a token prints a loud stderr warning — no hard failure (upstream proxies may legitimately terminate auth), but the intent is impossible to miss.

Client-side example (curl):

```bash
curl -N -H 'Authorization: Bearer <token>' http://host:3333/sse
```

#### JWT / JWKS validation (new in v1.16.0)

For real enterprise deployments, validate signed JWTs against your IdP's JWKS endpoint. Works with SAP XSUAA, Auth0, Okta, Azure AD, Google, Keycloak, AWS Cognito, and any other standards-compliant IdP.

```bash
npm install jose   # optional peer dep, only needed for JWT auth

saptarishi-llm mcp --http --host 0.0.0.0 \
  --jwks-url https://tenant.authentication.us10.hana.ondemand.com/token_keys \
  --jwt-issuer https://tenant.authentication.us10.hana.ondemand.com \
  --jwt-audience sb-my-cap-app!t12345
```

Behavior:
- Each request's `Authorization: Bearer <jwt>` is verified against the JWKS (signature, `exp`, `iat`, `nbf`, optional `iss` / `aud`).
- JWKS cached + auto-refreshed on unknown `kid` by `jose`.
- Any failure — bad signature, expired, wrong issuer, wrong audience, unknown key — collapses to a uniform 401. Never leaks *which* check failed (avoids oracle attacks; same rationale as bearer-token constant-time compare).

On BTP with an XSUAA service binding, use the values from `VCAP_SERVICES.xsuaa[0].credentials`:
- `--jwks-url  <url>/token_keys` (where `<url>` = credentials.url)
- `--jwt-issuer <url>`
- `--jwt-audience credentials.xsappname`

Programmatic use — build your own verifier for custom flows (introspection endpoints, mTLS metadata → role mapping, JWT + local RBAC check):

```js
const { createHttpTransport, createJwtVerifier } = require('@saptarishi/cds-plugin-llm');

const jwtVerifier = createJwtVerifier({ jwksUrl, issuer, audience });
createHttpTransport({
  server,
  authTokenVerifier: async (token) => {
    const claims = await jwtVerifier(token);
    if (!claims) return null;
    // Extra check: reject unless the client has the 'llm-user' scope
    if (!claims.scope?.includes('llm-user')) return null;
    return claims;
  },
});
```

#### Progress notifications (new in v1.12.0)

Long-running tools can push `notifications/progress` back to the client instead of leaving them hanging silently. Tool handlers grow an optional second argument — a `ctx` object with `reportProgress(current, total?)`:

```js
server.registerTool({
  name: 'batch_summarize',
  description: 'Summarize N documents',
  inputSchema: { type: 'object', properties: { docs: { type: 'array', items: { type: 'string' } } } },
  handler: async ({ docs }, ctx) => {
    const summaries = [];
    for (let i = 0; i < docs.length; i++) {
      summaries.push(await llm.chat({ /* ... */ }));
      ctx.reportProgress(i + 1, docs.length);
    }
    return summaries;
  },
});
```

When the client calls this tool with `_meta.progressToken: 'x'`, each `reportProgress(...)` becomes a server-pushed notification carrying that token. Delivered on **both** transports — stdio writes to the same stdout; HTTP+SSE pushes on the requesting session's SSE stream. Existing tools that ignore the second arg keep working (backwards-compatible).

#### Resource subscriptions (new in v1.17.0)

Complements the list-changed notifications from v1.13.0 with per-URI push updates. Use `list_changed` when the *set* of resources shifts, `resources/subscribe` when a *specific* resource's content changes.

```jsonc
// client -> server
{ "jsonrpc": "2.0", "id": 1, "method": "resources/subscribe",
  "params": { "uri": "prompt://summarize" } }

// server -> client (later, when that prompt is hot-reloaded)
{ "jsonrpc": "2.0", "method": "notifications/resources/updated",
  "params": { "uri": "prompt://summarize" } }
```

- **Per-connection state** — session A subscribing never affects session B. Cleaned up when the SSE stream / stdio pipe closes.
- **Both transports** — stdio writes updates to stdout on the reply stream; HTTP+SSE pushes to the subscribing session's stream only.
- **Unknown URIs rejected** at subscribe time so client typos surface immediately. Matches both static resources and template URIs (`prompt://{name}`, `provider://{kind}`).
- **`resources/unsubscribe`** is idempotent — safe to call for URIs you never subscribed to.

`--watch-prompts` now fires `resources/updated` for every subscribed `prompt://<name>` URI on hot-reload (in addition to the existing `prompts/list_changed` broadcast). Pin a client to a specific prompt and every edit refreshes without a `resources/read`.

For third-party transports / programmatic MCP setups, `MCPServer` exposes:

- `notifyResourceUpdated(uri)` — broadcast to every subscriber of `uri`. Silent no-op with none.
- `subscribedUris(prefix?)` — distinct URIs any connected client subscribed to, optionally prefix-filtered.

#### Per-session provider overrides (new in v1.18.0)

One `saptarishi-llm mcp` process, multiple named provider configs behind a single MCP endpoint. Clients pick which one to use per session or per tool call. Real enterprise use: one authenticated endpoint, different agents hit different backends — DevOps on `cheap`, compliance reviewer on `smart`, local dev on `local`. Credentials centralized server-side; agents never touch API keys.

```bash
saptarishi-llm mcp --http --host 0.0.0.0 \
  --providers-config ./providers.json
```

```jsonc
// providers.json  —  chmod 600 and keep out of git
{
  "cheap": {
    "kind": "groq",
    "model": "llama-3.1-8b-instant",
    "credentials": { "apiKey": "gsk_..." }
  },
  "smart": {
    "kind": "anthropic",
    "model": "claude-opus-4-7",
    "credentials": { "apiKey": "sk-ant-..." }
  },
  "local": {
    "kind": "ollama",
    "model": "qwen2.5:14b",
    "credentials": { "baseUrl": "http://localhost:11434" }
  }
}
```

**Resolution order per tool call**: `arguments.provider` → `sessionState.provider` (from `initialize._meta.provider`) → top-level default.

**Session default** — client declares once at initialize; every subsequent call on that session uses it:

```jsonc
{ "jsonrpc": "2.0", "id": 1, "method": "initialize",
  "params": { "protocolVersion": "2024-11-05",
              "_meta": { "provider": "smart" } } }
```

**Per-call override** — bypass the session default one call at a time:

```jsonc
{ "jsonrpc": "2.0", "id": 2, "method": "tools/call",
  "params": { "name": "chat",
              "arguments": { "prompt": "dispatch this", "provider": "cheap" } } }
```

**Discovery** — `list_providers` tool and `config://providers` resource both dump the alias list (kind + model per alias; credentials never returned). Unknown aliases surface as tool-result errors with the configured list so the model can self-correct on the next turn.

**Boot-time validation** — every alias is instantiated + `init()`'d at server startup. Bad credentials or unknown kinds die on `saptarishi-llm mcp`, not on the first client call.

## Response caching (new in v0.9.0)

Opt-in per-instance LRU cache with TTL. Skips tool-use calls (side-effects) and streaming (partial responses). Hits return the same `ChatResponse` shape with `cached: true` set.

```jsonc
{
  "cds": { "requires": { "llm": {
    "kind": "llm-groq",
    "modelId": "llama-3.3-70b-versatile",
    "responseCache": true                              // defaults: 5min TTL, 100 entries
    // or: "responseCache": { "ttlMs": 600000, "maxEntries": 500 }
  }}}
}
```

Cache key is a SHA-1 of the request's stable JSON representation (`model + maxTokens + system + messages + tools + format + thinking`). Requests with any of those fields differing miss the cache.

```js
const r1 = await llm.chat({ messages: [{ role: 'user', content: 'hi' }] });
r1.cached  // undefined  (miss + fresh call)

const r2 = await llm.chat({ messages: [{ role: 'user', content: 'hi' }] });
r2.cached  // true        (hit — no upstream call)

llm.responseCache.hits    // 1
llm.responseCache.misses  // 1
llm.responseCache.size()  // 1
```

Common wins:
- **Same PO summarized twice** (approver reopens the review) — instant, zero tokens
- **Batch classification** with duplicate inputs — deduplicates automatically
- **Load-testing** — dev-env queries hit cache after the first pass

## Semantic response cache (new in v2.7.0)

`semanticCache` is a middleware-form cache keyed by **embedding similarity**, not byte-equality — useful when the same question keeps showing up in slightly different wording. It's separate from the built-in `responseCache` (which is exact-match + LRU). Bring your own embedder and vector store; an in-memory linear-scan store is provided so you can wire it up with zero infra.

```js
const { semanticCache, inMemorySemanticStore } = require('@saptarishi/cds-plugin-llm');

llm.use(semanticCache({
  embedder: async (text) => (await llm.embed({ input: text })).embeddings[0],
  store:    inMemorySemanticStore({ maxEntries: 1000, ttlMs: 3600_000 }),
  threshold: 0.92,          // cosine similarity → hit
  keyPrefix: `tenant:${tenantId}:`,
  shouldCache: (_ctx, r) => r?.status !== 'error',
  onHit:  (i) => cds.log('llm:sc').info('hit', i),
}));
```

Highlights:
- **Fail-open.** Any exception from the embedder or store falls through to `next()` — a broken cache never takes the request path down.
- **Multi-tenant safe.** `keyPrefix` is post-filtered by the middleware even if the store ignores the hint, so hits cannot leak across tenants.
- **Swap the store for prod.** The interface is three async methods (`get`, `put`, `findSimilar`) — implement against pgvector, Redis, Pinecone, Weaviate, etc.
- **MCP resource** at `config://semantic-cache` exposes `hitRate`, hit/miss/store counts, threshold, and last similarity for live dashboards.

Composition rule: wrap `semanticCache` **outside** `bulkhead`/`retry` (cache hits shouldn't burn concurrency slots or retry budget) and **inside** `guardrails`/`promptInjectionGuard` (don't cache answers whose inputs were rejected).

## Fuzzy dedup (new in v2.38.0)

`fuzzyDedup` is **near-duplicate request detection** using cheap character-level similarity — Jaccard on character trigrams (default, no library deps) or normalized Levenshtein. Cache-like semantics: on match, returns the prior response without calling the provider. No embedder required, so it complements (rather than competes with) `semanticCache` (v2.7.0). Ideal for support-ticket dedup + spam prevention where users retype the same question with typos or slight rewording.

```js
const { fuzzyDedup, inMemoryFuzzyStore } = require('@saptarishi/cds-plugin-llm');

llm.use(fuzzyDedup({
  similarityKind: 'jaccard-trigram',   // or 'levenshtein'
  threshold:      0.85,
  store:          inMemoryFuzzyStore({ maxEntries: 5000, ttlMs: 24 * 3600_000 }),
  minKeyLength:   8,
  onHit:          (i) => cds.log('llm:fuzzy').info('near-dup', i),
}));
```

Fills the third slot in the dedup story alongside existing primitives:
- **`requestCoalescer`** (v2.8.0) — BYTE-IDENTICAL, IN-FLIGHT
- **`responseCache`** (v0.9.0) — BYTE-IDENTICAL, AT-REST
- **`semanticCache`** (v2.7.0) — EMBEDDING-SIMILAR (requires embedder)
- **`fuzzyDedup`** (v2.38.0, this) — CHARACTER-SIMILAR (no embedder)

Highlights:
- **Two-phase matching** — exact-key hit via `store.get` first (fast), then `store.findSimilar` fuzzy scan only as needed. `stats.exactHits` and `stats.fuzzyHits` are broken out separately so operators can see how much lift comes from typo-level matching vs plain repeats.
- **Two algorithms** — `jaccard-trigram` (O(n+m), scales to a few thousand chars) or `levenshtein` (O(n*m), catches char-level typos more precisely — cap prompt length in practice).
- **In-memory reference store** ships with LRU eviction (`maxEntries`) and optional TTL — same `{ get, put, findSimilar }` contract used by any custom store (Redis, PostgreSQL, etc.).
- **Multi-tenant safe** via `keyPrefix` — namespace one shared store across many dedup instances; post-filter enforces isolation even if the store ignored the hint.
- **`minKeyLength: 8`** default skips trivially-short prompts (`"hi"`, `"ok"`) where every prompt would collide.
- **Fail-open on store errors** — increments `storeErrors`, passes through; cache degrades to no-op rather than blocking.
- Standalone `jaccardTrigram(a, b)`, `normalizedLevenshtein(a, b)`, `levenshteinDistance(a, b)`, `trigrams(text)` exported for custom similarity strategies.
- MCP resource: `config://fuzzy-dedup`.

Composition rules: compose OUTSIDE `requestCoalescer` (v2.8.0) so byte-identical concurrent requests coalesce first, and OUTSIDE `responseCache` (v0.9.0) so exact repeats short-circuit before the fuzzy scan runs at all. Compose with `semanticCache` (v2.7.0) when both are available — fuzzyDedup catches typos + trivial rewordings cheaply; the semantic layer catches genuine paraphrases (higher recall, embedder cost). Order dedup → quota/cost primitives so fuzzy hits never accrue cost against `quotaManager` (v2.23.0) or `costOverrunPredictor` (v2.33.0).

## Empty-response detector (new in v2.37.0)

`emptyResponseDetector` catches broken model responses (empty string, whitespace-only, single-char replies, refusal patterns like `"I can't help with that"`) BEFORE they reach the caller. Common failure modes caught: provider hiccups (empty payload), safety-filter or guardrail evasion that returned `""`, or soft-refusals on legitimate requests.

```js
const { emptyResponseDetector } = require('@saptarishi/cds-plugin-llm');

llm.use(emptyResponseDetector({
  minChars:   10,
  onEmpty:    'retry',       // 'throw' | 'retry' | 'log'
  maxRetries: 1,
  onDetected: (i) => cds.log('llm:empty').warn('empty response', i),
}));
```

Distinct from other quality primitives:
- **`structuredOutputRepair`** (v2.9.0) — SCHEMA-driven repair
- **`responseRevision`** (v2.29.0) — RUBRIC-driven re-ask
- **`emptyResponseDetector`** (v2.37.0) — EMPTY / REFUSAL detection

Built-in refusal patterns (6 anchored-to-start regexes): `I can't/cannot/won't help/assist`, `I'm unable/not able/sorry to`, `Sorry, but I can't`, `I must decline/refuse`, `This request cannot be`, `As an AI assistant, I can't`. Anchored to start so mid-response mentions don't false-positive.

Highlights:
- **Custom detection**: pass `detectEmpty(result)` for full override; return `true`/`false` or `{ empty, reason }`. Return `null` to fall back to defaults.
- **Reason-aware retry prompt**: refusal detection asks the model to explain WHY rather than refuse outright; too-short asks for a substantive answer.
- **Per-reason stats** (`byReason`) — dashboard breakdowns across `too-short` / `whitespace` / `refusal-pattern` / custom.
- **Fail-safe custom detector** — exceptions fall back to default detection.
- MCP resource: `config://empty-response-detector`.

Compose with `responseRevision` (v2.29.0) OUTSIDE this detector — first check for emptiness; if substantive but low-scoring, then revise. Two-stage quality gate.

## Content-length gate (new in v2.36.0)

`contentLengthGate` is **pre-flight size validation**: reject (or truncate) prompts that exceed a per-model token budget BEFORE they hit the provider. Prevents 400 errors on over-limit contexts + saves tokens on obviously-too-large inputs. Distinct from time-based limiters (`deadline`, `gracePeriod`) — this is *size-based*.

```js
const { contentLengthGate } = require('@saptarishi/cds-plugin-llm');

llm.use(contentLengthGate({
  modelLimits: {
    'gpt-4o':          128_000,
    'gpt-4o-mini':     128_000,
    'claude-opus-4-7': 200_000,
    default:            64_000,
  },
  overageMode: 'truncate-oldest',   // 'throw' | 'truncate-oldest' | 'log'
  onOverage:   (i) => cds.log('llm:size').warn('over-limit', i),
}));
```

Three-way overage policy:
- **`'throw'`** (default) — raises `ContentLengthExceededError` (code `CONTENT_LENGTH_EXCEEDED`) with `.tokens`, `.chars`, `.limitTokens`, `.model`. Fail-fast.
- **`'truncate-oldest'`** — drops oldest messages until under limit, preserving `system` + latest user message by default (`preserveSystem` / `preserveLatestUser` opt-out).
- **`'log'`** — pass through unmodified with `onOverage` observability; let the provider decide (useful for shadow deployments before enforcing).

Highlights:
- **GPT-family token heuristic** by default (`char / 4`); users with real tokenizers pass `tokenEstimator(text) → tokens`.
- **Per-model limits** with `default` fallback; unknown models pass through unchecked with `unknownModelCount` counter for observability.
- **Original request restored** after truncation-mode calls — caller state unmodified, only the downstream provider saw the trimmed version.
- **`overageRate()`** returns `overageCount / totalCalls` for real-time tuning.
- MCP resource: `config://content-length-gate`.

Composition rule: put `contentLengthGate` **outside** providers so the check runs before any network work. Compose with `compactHistory` (v1.91.0) as complementary primitives — compact keeps multi-turn conversations bounded via LLM summarization; gate catches single-turn over-limit as the hard backstop.

## Latency histogram (new in v2.35.0)

`latencyHistogram` tracks per-dimension latency distributions using Prometheus-style bucketed counts (**bucket counts + sum + count** — not raw samples). Reports p50/p95/p99 percentiles cheaply. O(1) memory per dimension regardless of sample count. Complements `providerHealthAggregate` (v2.31.0 — single p95 verdict) with the **full percentile shape** + exportable Prometheus format.

```js
const { latencyHistogram } = require('@saptarishi/cds-plugin-llm');

const hist = latencyHistogram({
  dimensionsOf:    (ctx, result) => ({ model: result?.model ?? 'unknown' }),
  overThresholdMs: 15_000,             // p95 threshold
  onOverThreshold: (i) => cds.log('llm:slo').warn('p95 breach', i),
});
llm.use(hist);

// Query percentiles:
const p = hist.getPercentiles({ model: 'gpt-4o' });
// → { count: 145, sum: 92500, mean: 638, p50: 500, p95: 2500, p99: 5000 }

// Export for Prometheus scraping:
const prom = hist.prometheusHistograms('llm_latency_ms');
// llm_latency_ms_bucket{model="gpt-4o",le="500"} 87
// llm_latency_ms_bucket{model="gpt-4o",le="1000"} 120
// ...
// llm_latency_ms_sum{model="gpt-4o"} 92500
// llm_latency_ms_count{model="gpt-4o"} 145
```

Highlights:
- **Prometheus-canonical bucket layout** by default: `[5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000]` ms. Users with different SLOs pass a custom `buckets` array.
- **Percentile computation** via cumulative-count walk — O(buckets) regardless of sample count.
- **Filtered percentiles** — `getPercentiles({ model: 'gpt-4o' })` aggregates matching dimensions.
- **Custom percentile arrays** — `getPercentiles(filter, [10, 90, 99.9])` for tail-latency analysis.
- **Rising-edge over-threshold** callback rearms on recovery so subsequent breaches also fire.
- **Fail-safe**: dimensionsOf errors captured via `onError`; latency still recorded on downstream error paths.
- MCP resource: `config://latency-histogram`.

Compose with `providerHealthAggregate` (v2.31.0) — aggregate gives the healthy/degraded verdict; histogram gives the full percentile shape for diagnostics. Compose with `promMetrics` (v1.x) — expose `prometheusHistograms()` output alongside the shipped `llm_*` metrics for unified Grafana dashboards.

## User feedback aggregator (new in v2.34.0)

`userFeedbackAggregator` collects human ratings (thumbs up/down or 1-5 stars) per response and aggregates them by dimensions (prompt template, model, tenant) for real-time dashboards. **Two-part primitive**: middleware attaches `feedbackId` for later attribution; `submitFeedback(id, rating, meta?)` records the human rating once the user actually reacts.

```js
const { userFeedbackAggregator } = require('@saptarishi/cds-plugin-llm');

const feedback = userFeedbackAggregator({
  dimensionsOf: (ctx, result) => ({
    template: ctx.meta?.promptVersion?.name,
    model:    result?.model,
    tenant:   ctx.request.tenantId,
  }),
  ratingKind:   'binary',      // 'binary' | 'scale' | 'custom'
  windowMs:     7 * 24 * 3600_000,
});
llm.use(feedback);

// Later, when the user clicks 👍:
feedback.submitFeedback(response.feedbackId, 'up');

// Dashboard query:
feedback.getAggregate({ template: 'summarize' });
// → { totalRatings: 145, positive: 123, positiveRate: 0.848, ... }
```

Distinct from the mechanical eval primitives:
- **`scoreResponse`** (v2.4.0) — mechanical rubric scoring
- **`llmJudge`** (v1.84.0) — LLM-as-judge scoring
- **`promptExperiment`** (v2.20.0) — A/B test with automatic scoring
- **`userFeedbackAggregator`** (v2.34.0) — HUMAN feedback

Highlights:
- **Three rating kinds**: `'binary'` (thumbs), `'scale'` (1-N stars), `'custom'` (any number + user-supplied `positivityOf`).
- **Per-dimension aggregation** — `getAggregate(filter?)` for filtered stats; `snapshotByDimension('model')` for pivot-table dashboards.
- **Rolling window** (default 30 days) with lazy pruning. O(1) memory over long-running deployments.
- **Validation** — invalid ratings + unknown IDs rejected with `{ accepted: false, reason }` status object; never throws.
- **Multiple ratings per response** supported (thumbs vote + written comment via `meta`).
- MCP resource: `config://user-feedback`.

Compose with `promptVersionPin` (v2.30.0) — the pin annotates `ctx.meta.promptVersion` with `{ name, version, source }`; use those as `dimensionsOf` fields to attribute feedback per version. Roll out v4, see v4's positive rate lagging v3's, roll back via `pinFor`.

## Cost overrun predictor (new in v2.33.0)

`costOverrunPredictor` tracks spend against a **fixed calendar billing window** (typically end-of-month) and projects end-of-window spend based on the current burn rate. Warns rising-edge when the projected spend would exceed `targetUsd`. Distinct from `costForecast` (v2.1.0) — that projects a *rolling* window; this projects to a *fixed calendar boundary*.

```js
const { costOverrunPredictor, startOfMonth, endOfMonth } = require('@saptarishi/cds-plugin-llm');

llm.use(costOverrunPredictor({
  windowStart: () => startOfMonth(new Date()),
  windowEnd:   () => endOfMonth(new Date()),
  targetUsd:   1000,
  costOf:      (ctx, result) => {
    const inTok  = result.usage?.input_tokens  ?? 0;
    const outTok = result.usage?.output_tokens ?? 0;
    return (inTok / 1e6) * 0.15 + (outTok / 1e6) * 0.60;
  },
  warnAtRatio: 0.85,
  onWarn:      (i) => cds.log('llm:cost').warn('overrun-projected', i),
  onExhausted: (i) => cds.log('llm:cost').error('overrun-imminent', i),
}));
```

Fills the cost stack:
- **`costBudget`** (v1.x) — GLOBAL cap (hard block)
- **`quotaManager`** (v2.23.0) — PER-USER cap
- **`costForecast`** (v2.1.0) — ROLLING-window projection
- **`costOverrunPredictor`** (v2.33.0) — CALENDAR-window projection
- **`costAwareRouter`** (v2.10.0) — cheap-first with quality escalation

Highlights:
- **Linear projection** — `projected = (spent / elapsed) × fullWindow`. Simple + effective for steady-state workloads.
- **Rising-edge warn** at `warnAtRatio × targetUsd` (default 0.85). Separate **rising-edge exhausted** at `targetUsd` so alerting distinguishes "on track to overspend" from "definitely overspending."
- **Auto window rollover** — when `windowStart()` returns a new timestamp, spend + rising-edge state reset automatically. No cron jobs needed.
- **`minSampleSize` gate** (default 20) — prevents projection noise during the first minutes of a window.
- **`projection()`** returns full live state for dashboards + ops UIs.
- **Six calendar helpers exported**: `startOfMonth`, `endOfMonth`, `startOfDay`, `endOfDay`, `startOfQuarter`, `endOfQuarter`.
- MCP resource: `config://cost-overrun-predictor`.

Compose with `quotaManager` (v2.23.0) — predictor gives early warning ("you'll blow the monthly budget"); quota gives hard enforcement ("this user hit their cap"). Both signals feed the same alerting pipeline.

## Response signing (new in v2.32.0)

`responseSigning` HMAC-signs the LLM response before delivering to the caller. Downstream consumers (SAP Event Mesh, message queues, service mesh hops) can verify authenticity + integrity offline with the shared secret. Complements `requestSigning` (v2.21.0):
- **`requestSigning`** — signs OUTBOUND requests + emits per-call receipts (audit trail)
- **`responseSigning`** — signs the RETURNED response so it can be verified wherever it later travels

```js
const { responseSigning, verifyResponseSignature } = require('@saptarishi/cds-plugin-llm');

llm.use(responseSigning({
  secret:    process.env.LLM_RESPONSE_HMAC_KEY,
  algorithm: 'sha256',
}));

// Downstream (possibly in a different process/service):
const { valid, reason } = verifyResponseSignature(result, secret);
// { valid: true, reason: null }   or   { valid: false, reason: 'hash-mismatch' }
```

The signature envelope attaches as `result.signature = { hash, sig, algorithm, ts }`. Consumers who don't care about signing just ignore the field; it doesn't interfere with `result.text` / `result.data` / `result.toolCalls`.

Highlights:
- **Deterministic** — canonical serialization uses a stable-key-order JSON stringifier so identical results always produce identical hashes across restarts + processes.
- **Standalone verifier** — `verifyResponseSignature(result, secret, options?)` usable in any downstream service without the middleware runtime.
- **Fail-safe** — signing errors are captured via `onError`; the underlying response is still returned unsigned. Non-object results (plain strings) skip signing without error.
- **Three algorithms**: `sha256` / `sha384` / `sha512`. Frozen list: `RESPONSE_SIGNING_ALGORITHMS`.
- MCP resource: `config://response-signing`.

Composition rule: place `responseSigning` at the **outermost layer** so the signed result is what actually returns to the caller (post any repair, cache, hedge). Any middleware wrapping around it would either re-sign or invalidate the signature.

## Provider health aggregate (new in v2.31.0)

`providerHealthAggregate` combines multiple health signals per provider into a **unified score** — routing decisions can consult one metric instead of stitching together five. Instruments every call, records latency + success/error per provider, computes a rolling error rate + p95 latency, emits `onDegraded` / `onRecovered` transitions with hysteresis via `minSampleSize`.

```js
const { providerHealthAggregate } = require('@saptarishi/cds-plugin-llm');

const health = providerHealthAggregate({
  providerOf:          (ctx, result) => result?.model?.split('/')[0] ?? 'unknown',
  windowMs:            60_000,
  errorRateThreshold:  0.10,           // >10% errors → degraded
  latencyP95Threshold: 15_000,          // >15s p95 → degraded
  minSampleSize:       10,             // don't judge until enough data
  onDegraded:  (i) => cds.log('llm:health').error('degraded', i),
  onRecovered: (i) => cds.log('llm:health').info('recovered', i),
});
llm.use(health);

// Later, in routing logic:
const s = health.getHealth('openai');
// → { healthy: true, score: 0.92, errorRate: 0.02, latencyP95Ms: 3200, samples: 145, ... }
```

Complements the four other health primitives:
- **`providerHealthProbe`** (v1.62.0) — PROACTIVE liveness pings
- **`circuitBreaker`** (v1.x) — REACTIVE per-error trip
- **`adaptiveBulkhead`** (v1.61.0) — latency-driven concurrency tuner
- **`providerHealthAggregate`** (v2.31.0) — UNIFIED score across all signals

Highlights:
- **Weighted composite** (default: 60% error rate + 40% p95 latency; optional 20% breaker-state weight when `breakerFor` is wired). Score in `[0, 1]` where 1 = fully healthy.
- **Sample-size gate** prevents cold-start false-degrades — verdict is not computed until `minSampleSize` samples accumulate.
- **Transition callbacks fire on state changes only** (rising / falling edges), not per-call — perfect for alerting.
- **Rolling window** with lazy pruning — O(1) memory per provider over long-running processes.
- **`snapshotAll()`** / **`listProviders()`** for dashboards.
- MCP resource: `config://provider-health-aggregate`.

Composition rule: put `providerHealthAggregate` **outside** providers so it observes every call. It's a passive instrumentation layer — never modifies the request path. Use `getHealth()` in your `chatWithFallback` / `regionFailover` selector to skip degraded providers proactively.

## Prompt version pinning (new in v2.30.0)

`promptVersionPin` is version selection for prompt templates: **rollback-friendly** for canary deployments where 5% of traffic pins to v3 while 95% runs v4, or per-session rollback for a user who reported a bad response on the new version. Ships with a lightweight `PromptVersionRegistry` class; bring-your-own resolver if you want to back the registry with Git commit SHAs, a database, or `gitPromptRegistry` (v2.1.0).

```js
const { promptVersionPin, PromptVersionRegistry } = require('@saptarishi/cds-plugin-llm');

const registry = new PromptVersionRegistry();
registry.register('summarize', 1, (vars) => ({
  messages: [{ role: 'user', content: `Summarize (v1): ${vars.text}` }],
}));
registry.register('summarize', 2, (vars) => ({
  messages: [{ role: 'user', content: `Summarize concisely (v2): ${vars.text}` }],
}));

llm.use(promptVersionPin({
  resolveTemplate: (name, version) => registry.getVersion(name, version),
  latestVersionOf: (name) => registry.latestVersion(name),
  templateRefOf:   (ctx) => ctx.request.templateRef,
  pinFor:          (ctx, name) => sessionPins[ctx.request.sessionId]?.[name],
  applyTemplate:   (req, tpl, ctx) => ({ ...req, ...tpl(req.vars ?? {}) }),
  onPin: (i) => cds.log('llm:pin').info(i),
}));

// Callers set:
llm.chat({ templateRef: { name: 'summarize' }, vars: { text: '...' } });
```

Resolution priority (highest first):
1. **`pinFor(ctx, name)`** — per-request pin (canary / rollback)
2. **`templateRef.version`** — explicit caller-supplied version
3. **`latestVersionOf(name)`** — falls back to registry's latest

Audit metadata written to `ctx.meta.promptVersion = { name, version, source }` — downstream logs, MCP resources, and dashboards see exactly which template version was used per call.

Highlights:
- **Bring-your-own registry** — the middleware just needs `resolveTemplate` + `latestVersionOf` functions. Wire in `gitPromptRegistry` (v2.1.0) with commit SHAs as versions, a database, whatever.
- **Fail-open** — missing templates or unknown versions fall through to `next()` without modifying the request. Never blocks the request path.
- **Per-template usage counts** (`stats.byTemplate.summarize = { '1': 12, '2': 45 }`) + `pinRate()` for canary-progress monitoring.
- MCP resource: `config://prompt-version-pin`.

Compose with `promptExperiment` (v2.20.0) — experiment picks the variant (name), pin locks a version of that variant. Rollback while A/B testing.

## Response revision loop (new in v2.29.0)

`responseRevision` is a quality-driven revision loop. When the response scores below threshold, this middleware automatically re-asks the **same model** with the low-scoring response + rubric feedback inline, up to `maxRevisions` times. Returns the **best-scoring** response across all attempts even if none pass the threshold.

```js
const { responseRevision, scoreResponse } = require('@saptarishi/cds-plugin-llm');

llm.use(responseRevision({
  scorer: (result) => {
    const report = scoreResponse(result, { rubric: myRubric });
    return {
      score: report.score,
      feedback: report.results
        .filter((r) => !r.pass)
        .map((r) => `- ${r.check}: ${r.reason}`)
        .join('\n'),
    };
  },
  scoreThreshold: 0.8,
  maxRevisions:   2,
  onRevision: (i) => cds.log('llm:revise').info('revision', i),
}));
```

Distinct from the three other quality primitives:
- **`structuredOutputRepair`** (v2.9.0) — SCHEMA-driven repair
- **`costAwareRouter`** (v2.10.0) — escalates to a DIFFERENT model
- **`consensusVoting`** (v2.5.0) — fans out to N models in parallel
- **`responseRevision`** (v2.29.0) — iterates on the SAME model with rubric feedback

Highlights:
- **Flexible scorer**: return `number` in `[0, 1]` OR `{ score, feedback }`. Feedback (if present) is injected into the revision prompt.
- **Best-effort fallback**: even when all revisions fail, returns the highest-scoring attempt (not the last). Prevents the "kept re-asking and got progressively worse" tail.
- **Fail-safe scorer**: exceptions or non-numeric returns treated as unscorable → return the first result unmodified. Never blocks the request path.
- **`avgRevisions()`** and **`passRate()`** for real-world tuning of `maxRevisions` / `scoreThreshold`.
- Default `applyRevision` appends previous assistant reply + user revision request — matches the standard multi-turn revision pattern most models understand.
- MCP resource: `config://response-revision`.

Composition rule: place `responseRevision` **outside** `responseCache` / `semanticCache` (revisions shouldn't be cached mid-loop). Compose with `costAwareRouter` (v2.10.0) *underneath* — revise on the cheap model first; if still low, escalate to premium as the fallback path.

## Grace period / soft deadline (new in v2.28.0)

`gracePeriod` complements the shipped `deadline` (v1.x hard timeout) with a **soft deadline** that fires a warning callback while the request keeps running. Useful for SLO monitoring: catch tail-latency spikes early, *before* they become full timeouts.

```js
const { gracePeriod } = require('@saptarishi/cds-plugin-llm');

llm.use(gracePeriod({
  softMs: 5000,          // warn at 5s
  hardMs: 15000,          // kill at 15s
  onSoftDeadline: (i) => cds.log('llm:slo').warn('slow', i),
  onHardDeadline: (i) => cds.log('llm:slo').error('killed', i),
}));
```

Three modes:
- **`softMs` alone** → warning only, no kill (pure SLO monitoring)
- **`hardMs` alone** → equivalent to the shipped `deadline`
- **`softMs` + `hardMs`** → warn at soft, kill at hard (SLO + safety net)

Highlights:
- **AbortSignal integration** — fresh `AbortController` per call on `ctx.signal` (opt-out via `attachAbortSignal: false`). Aborts on hard deadline so cancellation-aware providers early-exit instead of continuing to burn tokens after the timeout fires. Original ctx.signal restored after the call.
- **`GracePeriodExhaustedError`** (code `GRACE_PERIOD_EXHAUSTED`) with `.elapsedMs`, `.hardMs`, `.softMs`.
- **Buckets** — `completedUnderSoft` vs `completedOverSoft` for real-world SLO tuning. `softDeadlineRate()` returns the fraction of calls that crossed the soft line.
- MCP resource: `config://grace-period`.

Compose with `speculativeHedge` (v2.12.0) — use `onSoftDeadline` as a signal to fire a late hedge ("primary is probably slow, start a backup"). Compose with `promMetrics` — feed `onSoftDeadline` into a histogram bucket for real-time p95/p99 tracking.

## Batch aggregator (new in v2.27.0)

`batchAggregator` is window-based pooling of concurrent LLM calls. Waits up to `batchWindowMs` for a batch to fill (or hits `maxBatchSize`), fires **one** upstream call, then splits the response back to individual callers. Real cost saver for high-throughput fan-out patterns with uniform structure (classification, embedding, translation) where N similar prompts can be concatenated into 1 request and the response is an ordered array.

```js
const { batchAggregator } = require('@saptarishi/cds-plugin-llm');

llm.use(batchAggregator({
  batchWindowMs: 50,
  maxBatchSize:  10,
  batchable:     (ctx) => ctx.request.batchKey === 'classify-support-ticket',

  // User owns the concatenation logic:
  aggregateRequests: (batch) => ({
    messages: [{ role: 'user', content:
      'Classify each ticket (respond as a JSON array of labels):\n' +
      batch.map((b, i) => `${i+1}. ${b.ctx.request.messages[0].content}`).join('\n')
    }],
    format: { type: 'array', items: { enum: ['billing', 'bug', 'question'] } },
  }),
  // User owns the split logic:
  splitResponse: (result, batch) => {
    const labels = result.data;
    return batch.map((b, i) => ({ ...result, data: labels[i], text: labels[i] }));
  },

  onBatch: (i) => cds.log('llm:batch').info('flushed', i),
}));
```

Distinct from other batching primitives:
- **`requestCoalescer`** (v2.8.0) — dedups IDENTICAL requests
- **`runBatch`** / **`waitForBatch`** (v1.79.0) — offline batch-API workflows (24-hour turnaround)
- **`batchAggregator`** (v2.27.0) — ONLINE pooling of DIFFERENT requests across a short window

Highlights:
- **`batchable(ctx)`** predicate opts calls in — non-uniform prompts pass straight through.
- **`batchKeyOf(ctx)`** partitions pending batches (e.g., per model, per user, per tenant) — members with different keys batch independently.
- **Fail-loud contract** — if `splitResponse` returns a wrong-count or non-array, all batch members fail with `BatchAggregationError` (code `BATCH_AGGREGATION_FAILED`) so bugs surface immediately.
- **`pendingCount()`, `pendingKeys()`, `avgBatchSize()`** for tuning.
- MCP resource: `config://batch-aggregator`.

Composition rule: put `batchAggregator` **outside** `semanticCache` / `responseCache` / `requestCoalescer` (batching only happens on cache misses) and **inside** `bulkhead` (one batch = one slot).

## Client-side rate limiter (new in v2.26.0)

`clientSideRateLimit` is a proactive N-per-window throttler. Blocks/queues LLM calls to stay under a configured rate rather than letting the provider 429 you and relying on after-the-fact recovery. Closes the reactive-vs-proactive rate-limit story:

- **`retryOnRateLimit`** (v1.x) — REACTS to 429 by waiting + retrying
- **`adaptiveRateLimit`** (v2.6.0) — REACTS to headers by shrinking bulkhead
- **`clientSideRateLimit`** (v2.26.0) — PROACTIVELY shapes traffic to never hit the cap

```js
const { clientSideRateLimit } = require('@saptarishi/cds-plugin-llm');

llm.use(clientSideRateLimit({
  strategy:       'token-bucket',
  rate:           10,          // requests per second
  burst:          20,          // bucket capacity
  keyOf:          (ctx) => ctx.request.userId ?? 'global',
  queueTimeoutMs: 30_000,
  onTimeout:      (i) => cds.log('llm:rate').warn('timeout', i),
}));
```

Two strategies:
- **`'token-bucket'`** — classic Nginx-style: refill at `rate` per second up to `burst` capacity; consume 1 token per call. Best for bursty traffic where short spikes are OK.
- **`'sliding-window'`** — strict N-per-`windowMs` count. Best when you have a documented provider quota like "100 requests/min" and want exact enforcement.

Highlights:
- **Per-key limits** via `keyOf(ctx)` — separate token buckets/windows per tenant, per model, per user. Null/empty key falls to the `'global'` bucket.
- **FIFO queue with timeout** — over-limit calls wait up to `queueTimeoutMs`; then reject with `RateLimitTimeoutError` (code `RATE_LIMIT_QUEUE_TIMEOUT`) carrying `.rateLimitKey`, `.waitedMs`, `.queueTimeoutMs`.
- **Per-key drain scheduler** ticks only when there are waiters — idle keys consume zero CPU.
- **`snapshotKeys()`** returns per-key state (tokens or timestamp count, queue depth) for real-time dashboards. `avgWaitMs()` for tuning burst/window/timeout.
- MCP resource: `config://client-rate-limit`.

Composition rule: put `clientSideRateLimit` **outside** providers so throttling happens before any network work. Compose with `retryOnRateLimit` (inside) as defense in depth — client-side shapes traffic; retry recovers if the provider's actual quota drifted lower than expected.

## Multimodal attachment router (new in v2.25.0)

`multimodalRouter` inspects the request's message content blocks, detects which media types are present (text, vision, pdf, audio, video), and routes to the model configured for that capability set. Fills the "how do I automatically pick the right model for a vision request vs. a text request?" gap that the four other routers don't cover.

```js
const { multimodalRouter } = require('@saptarishi/cds-plugin-llm');

llm.use(multimodalRouter({
  routes: {
    'text':          { model: 'openai/gpt-4o-mini' },
    'vision':        { model: 'openai/gpt-4o' },
    'text+vision':   { model: 'openai/gpt-4o' },
    'pdf':           { model: 'anthropic/claude-opus-4-7' },
    'audio':         { model: 'openai/whisper-1' },
    'pdf+vision':    { model: 'anthropic/claude-opus-4-7' },
  },
  fallbackKey: 'text',
  onRoute:    (i) => cds.log('llm:mm').info('routed', i.key),
  onFallback: (i) => cds.log('llm:mm').warn('no route for', i.detectedKey),
}));
```

Detection recognizes both Anthropic and OpenAI shapes:
- `{ type: 'image' }` / `{ type: 'image_url' }` / `{ type: 'image_base64' }` → **vision**
- `{ type: 'pdf' }` / `{ type: 'document', mimeType: 'application/pdf' }` → **pdf**
- `{ type: 'audio' }` / `{ type: 'input_audio' }` → **audio**
- `{ type: 'video' }` → **video**
- string content or `{ type: 'text' }` blocks → **text**
- MIME sniffing on `document`/`file` blocks (`image/*` → vision, `audio/*` → audio, etc.)

Complements the four other routers:
- **`modelRouter`** — static keyword rules
- **`semanticRouter`** — embedding-based routing
- **`costAwareRouter`** — quality-driven escalation
- **`multimodalRouter`** — capability-aware routing

Compose them: multimodal picks the capability tier upfront; cost-aware escalates within that tier if the response scores low.

Highlights:
- **Canonical sorted-set keys** (`'pdf+vision'`, not `'vision+pdf'`). Middleware validates + rejects unsorted keys at construction so config mistakes fail loud.
- **Respects capability matrix** — routes are just config; if you route vision to a text-only model, `capabilities(llm)` (v2.3.0) flags the mismatch and `preflight` refuses to boot.
- **Fail-open** on detection error — a throwing `detectAttachments` falls through unmodified.
- MCP resource: `config://multimodal-router` (route table + distribution + full stats).

## Streaming chunk aggregator (new in v2.24.0)

`streamAggregator` buffers per-token stream chunks and emits **aggregated** chunks either when the buffer reaches `minChars` (default 20) OR when it's been idle for `maxIdleMs` (default 100ms). Reduces UI cursor jitter when the provider emits per-character chunks — a common quirk of some OpenAI-compat endpoints and local models via Ollama.

```js
const { streamAggregator, streamThrottle } = require('@saptarishi/cds-plugin-llm');

llm.use(streamThrottle({ maxTokensPerSecond: 40 }));   // outer — paces cadence
llm.use(streamAggregator({
  minChars:  20,      // flush when buffer >= 20 chars
  maxIdleMs: 100,     // OR after 100ms idle
  onFlush:   (i) => cds.log('llm:agg').debug('flush', i),
}));
```

Distinct from the shipped `streamThrottle` (v1.97.0):
- **`streamThrottle`** — PACES existing chunks with a delay (throttle bursts)
- **`streamAggregator`** — COMBINES chunks (fewer to render)

Use both together: aggregator produces fewer chunks; throttle paces them at a smooth cursor speed.

Highlights:
- **`Promise.race`** between source chunk and idle timer — flushes as soon as EITHER condition fires. Timer is `unref()`'d so it doesn't hold the event loop.
- **Terminal chunk detection** — `done`, `isDone`, `type: 'done'`, `finish_reason`, `stopReason` all recognized as end-of-stream markers. Buffer flushed first, then terminal chunk passes through untouched so the `wrapStreamCompletion` (v1.72.0) tracker fires with the correct `doneChunk`.
- **Non-text chunks pass through** — tool call deltas, metadata chunks, etc. bypass the buffer. Ordering preserved: buffer flushes BEFORE the non-text chunk emits.
- **Preserves stream completion** — `hasStreamCompletion` still detects the wrapped stream; `.onComplete()`, `.completedInfo`, `.isCompleted` all re-exposed.
- **Custom shapes** — `extractText(chunk)` for provider-specific chunk formats (e.g., OpenAI's `delta.content`); `makeChunk(text)` to change the emitted-chunk shape.
- **`reductionRatio()`** returns `1 - (emitted / source)`. Higher = more coalescing happening — instrumented for tuning `minChars` / `maxIdleMs`.
- MCP resource: `config://stream-aggregator`.

## Per-user quota manager (new in v2.23.0)

`quotaManager` is per-user (or per-tenant) USD quota with sliding-window tracking, rising-edge warning thresholds, and hard cap with configurable grace period. Real ask for SaaS deployments that need to bill or throttle by usage. Blocks calls **before** any real work with `QuotaExhaustedError`, so an over-quota user doesn't consume a slot or a retry budget.

```js
const { quotaManager, inMemoryQuotaStore } = require('@saptarishi/cds-plugin-llm');

llm.use(quotaManager({
  keyOf:  (ctx) => ctx.request.userId ?? 'anon',
  store:  inMemoryQuotaStore({ maxKeys: 10_000 }),
  costOf: (ctx, result) => {
    // Compute USD from result.usage using your provider pricing.
    const inputCost  = (result.usage?.input_tokens  ?? 0) * 0.15 / 1e6;
    const outputCost = (result.usage?.output_tokens ?? 0) * 0.60 / 1e6;
    return inputCost + outputCost;
  },
  quotas: {
    default:      { limitUsd: 10 },      // most users
    'user-vip':   { limitUsd: 1000 },     // VIP tier
    'user-power': { limitUsd: 100 },
  },
  windowMs: 30 * 24 * 3600_000,           // 30-day rolling window
  warnThresholds: [0.5, 0.8, 0.95],
  gracePeriodRatio: 0.02,                 // allow 2% overshoot
  onWarn:      (i) => cds.log('llm:quota').warn('threshold', i),
  onExhausted: (i) => cds.log('llm:quota').error('exhausted', i),
}));
```

Complements the shipped cost primitives — pick the right layer for the job:
- **`costBudget`** — GLOBAL cap across all users
- **`costGuard`** — per-call cap (single-call safety net)
- **`costForecast`** (v2.1.0) — burn-rate projection (early warning)
- **`costAwareRouter`** (v2.10.0) — quality-driven tier escalation
- **`quotaManager`** (v2.23.0) — per-user fairness enforcement

Highlights:
- **Pluggable store** — three async methods (`get`, `add`, optional `setLastWarnLevel` / `reset`). Ships with `inMemoryQuotaStore` (LRU eviction) as zero-infra default; swap for Redis/DynamoDB/HANA for persistence across restarts.
- **Grace period** (default 2%) — prevents "you were charged $10.01 for a $10 quota" complaints while keeping the cap meaningful. Set to 0 for a strict cap.
- **Rising-edge warnings** — each threshold fires once as usage crosses it upward. Reset naturally as samples age out of the sliding window.
- **`getUsage(key)`** returns `{ usageUsd, limitUsd, utilization, samplesInWindow }` for real-time dashboards and admin UIs.
- **`resetKey(key)`** manually clears one user's usage — useful for ops overrides (compensation, quota rebase).
- **Fail-open** — a broken store never takes the request path down.
- MCP resource: `config://quota-manager`.

Composition rule: put `quotaManager` **outside** `bulkhead` / `retryOnRateLimit` / providers — the quota check should short-circuit BEFORE any real work. Compose with `fairShareScheduler` (v2.14.0) for the full multi-tenant story: fair-share enforces per-tenant *concurrency*, quota manager enforces per-tenant *spend*.

## Auto tool chain (new in v2.22.0)

`autoToolChain` is the cascading tool-call loop that every agentic workflow needs. When the model returns `tool_calls`, this middleware automatically runs the tools via registered handlers, feeds tool results back as `tool` messages, re-invokes the chain, and loops until the model returns a final answer — or hits a safety cap. Frees callers from writing the tool-invocation-loop boilerplate for every agentic workflow.

```js
const { autoToolChain, functionCallArbitrator } = require('@saptarishi/cds-plugin-llm');

// Compose with functionCallArbitrator (v2.18.0) for allowlist + schema safety:
llm.use(functionCallArbitrator({ tools: [
  { name: 'lookup_customer', input_schema: {...} },
  { name: 'send_email',      input_schema: {...} },
], onInvalid: 'strip' }));

llm.use(autoToolChain({
  handlers: {
    lookup_customer: async (input) => ({ id: input.id, name: 'Alice', email: 'alice@x.com' }),
    send_email:      async (input) => ({ sent: true, to: input.to }),
  },
  maxDepth:          10,          // safety cap
  detectCycles:      true,        // reject same tool + same input twice
  handleUnknownTool: 'throw',     // 'throw' | 'skip' | 'error-back'
  onToolCall:        (i) => cds.log('llm:chain').info('tool', i),
  onChainComplete:   (i) => cds.log('llm:chain').info('done at depth', i.depth),
}));
```

Highlights:
- **`maxDepth`** (default 10) bounds chain depth. Exceeding throws `MaxDepthExceededError` (code `MAX_TOOL_DEPTH_EXCEEDED`).
- **Cycle detection** (default on) — stable-JSON hash of `(toolName, input)` catches the model calling the same tool with same input twice in a row. Throws `ToolChainCycleError` (code `TOOL_CHAIN_CYCLE`).
- **Three-way policy on unknown tools**: `'throw'` (default `UnknownToolError`), `'skip'` (drop), `'error-back'` (send `{error: "unknown tool"}` to model so it can recover).
- **Handler errors sent back to model** — a handler that throws emits an `{error: message}` tool result instead of propagating. Matches the OpenAI / Anthropic pattern for agentic loops.
- **Correct multi-turn construction** — assistant tool_use turn appended before tool results, one `{role: 'tool', tool_call_id, content}` message per call. Handles the message-stack accumulation across hops automatically.
- **`avgChainDepth()` + `maxObservedDepth`** — instrumented for real-world tuning.
- MCP resource: `config://auto-tool-chain`.

Composition rule: put `functionCallArbitrator` (v2.18.0) **outside** this middleware — the arbitrator validates + allowlists tool calls; this middleware runs them. Together: only vetted + schema-validated calls trigger a handler. Place **outside** `structuredOutputRepair` (v2.9.0) so repair operates on the FINAL response, not intermediate tool-request responses.

## Request signing / HMAC receipts (new in v2.21.0)

`requestSigning` cryptographically signs each outbound LLM call with HMAC-SHA256/384/512 and emits a **hash-chained receipt** on the response. Offline auditors can later verify with a shared secret that request/response pairs weren't tampered with between capture and audit. Strong compliance differentiator for SAP financial / healthcare / regulated workloads.

```js
const { requestSigning, verifyReceiptChain } = require('@saptarishi/cds-plugin-llm');

const receipts = [];
llm.use(requestSigning({
  secret:     process.env.LLM_HMAC_KEY,   // Buffer or hex string
  algorithm:  'sha256',                    // 'sha256' | 'sha384' | 'sha512'
  onReceipt:  (r) => receipts.push(r),     // persist to HANA/S3/etc
  includeRequestBody: false,               // opt-in — receipts stay small
}));

// Later, offline:
const { valid, brokenIdx, reason } = verifyReceiptChain(receipts, secret);
// { valid: true, brokenIdx: -1, reason: null }
```

Complements the shipped `sensitiveDataAudit` (v1.96.0):
- `sensitiveDataAudit` — immutable hash-chained audit LOGS (what happened)
- `requestSigning` — cryptographically-signed RECEIPTS keyed to a secret only the auditor can verify (non-repudiation)

Together they give the full compliance story.

Highlights:
- **Chain continues across errors** — failed calls emit receipts with `isError: true`. Compliance requires audit of attempts, not just successes.
- **Detects tampering + reordering** — each receipt is signed over `(requestHash + responseHash + prevReceiptHash)`, and `prevReceiptHash` is the SHA hash of the prior receipt's `sig`. Tampering with any field breaks the chain from that point forward; reordering breaks it too.
- **Canonical serialization** excludes volatile fields (credentials, retries config) so the same logical call produces the same signature across restarts. Users can supply custom `canonicalizeRequest` / `canonicalizeResponse` if needed.
- **`chainCursor()`** returns `{ index, prevSigHash }` — useful for live observability + persisting chain state across restarts (survives `reset()`).
- MCP resource: `config://request-signing`.

Composition rule: place `requestSigning` **outside** `providerLoadBalancer` (v2.17.0) — the signature should be over the canonical request, not the credential-applied one (the canonical serializer already excludes credentials so this is defensive).

## Prompt A/B experiment framework (new in v2.20.0)

`promptExperiment` runs N variants of a prompt against each other in **production traffic**, captures per-variant score + latency + cost distributions, and declares a statistically-defensible winner via 95% CI overlap check. Complements the shipped offline eval primitives (`llmJudge`, `promptRegression`, `scoreResponse`, `consensusVoting`) with the online production-traffic half of the eval story.

```js
const { promptExperiment, scoreResponse } = require('@saptarishi/cds-plugin-llm');

llm.use(promptExperiment({
  name: 'invoice-summarization-2026Q3',
  variants: [
    { name: 'control', weight: 5 },
    { name: 'terse',   weight: 3, apply: (req) => ({ ...req, system: 'Be terse. Bullets only.' }) },
    { name: 'chatty',  weight: 2, apply: (req) => ({ ...req, system: 'Be friendly + detailed.' }) },
  ],
  splitKeyOf:    (ctx) => ctx.request.userId ?? ctx.request.sessionId,
  scorer:        (result) => scoreResponse(result, { rubric: myRubric }).score,
  costEstimator: (result) => (result.usage?.output_tokens ?? 0) * 0.00001,
  onWinner: (i) => cds.log('llm:exp').warn('winner!', i),
}));

// Later, check the current winner:
const w = experiment.getWinner();
// { winner: 'terse', status: 'confident',
//   top: 'terse',   topScore: 0.87, topCI: [0.82, 0.92],
//   runnerUp: 'control', runnerUpScore: 0.71, runnerUpCI: [0.66, 0.76],
//   variants: [...] }
```

Highlights:
- **Weighted consistent-hash assignment** — same `splitKey` always maps to the same variant, so users experience the same variant across restarts, in-flight requests, and session reloads. Distinct experiments assign independently (a user in "control" of Experiment A may be in "v2" of Experiment B).
- **Welford's online algorithm** for score / latency / cost mean + variance. O(1) memory regardless of sample count — safe to run indefinitely without accumulating storage.
- **95% CI winner detection** with three status values:
  - `'insufficient-samples'` — fewer than `minSampleSize` (default 30) per variant
  - `'inconclusive-overlap'` — CIs of top two variants overlap; difference is within noise
  - `'confident'` — top CI doesn't overlap runner-up; statistically defensible
- **Multiple experiments compose** — just declare each with a different `name` and stack them in the chain. Each has its own MCP resource (`config://prompt-experiment/${name}`).
- Standalone helpers `hash32(str)`, `welfordUpdate`, `welfordStats`, `ciAroundMean` exported for custom analytics outside the middleware.

Composition rule: put `promptExperiment` **outside** `semanticCache`/`responseCache` (variants must not share cache entries; otherwise treatment leaks across arms). Use the session ID as `splitKeyOf` when composed with `sessionContextStore` so the same session always sees the same variant across turns (avoids mixed-treatment noise within a conversation).

## Session context store (new in v2.19.0)

`sessionContextStore` is multi-turn memory management for chat apps. On each call it prepends per-session message history to the request, appends the new user + assistant turns to the store after the response, and prunes to a sliding window (oldest-drop or synthetic-summary replacement). Solves the "how do I remember what the user said 3 turns ago without blowing the context budget" problem without hand-rolling per-app state.

```js
const { sessionContextStore, inMemorySessionStore } = require('@saptarishi/cds-plugin-llm');

llm.use(sessionContextStore({
  sessionOf:   (ctx) => ctx.request.sessionId,
  store:       inMemorySessionStore({ maxSessions: 10_000, ttlMs: 3600_000 }),
  maxMessages: 20,
  // Optional: summarize dropped messages instead of just dropping them.
  pruneStrategy: 'summarize',
  summarizer: async (dropped) => (await llm.chat({
    messages: [{ role: 'user', content: 'Summarize briefly:\n' + JSON.stringify(dropped) }],
  })).text,
  onSessionHit:  (i) => cds.log('llm:session').debug('hit',  i),
  onSessionMiss: (i) => cds.log('llm:session').debug('miss', i),
}));
```

Highlights:
- **Pluggable store** — three async methods (`get`, `put`, `append`) + optional `delete` + `size`. Ships with `inMemorySessionStore` (LRU eviction, optional TTL) as a zero-infra default; swap for pgvector/Redis/DynamoDB in prod using the same interface.
- **System-prompt preservation** — a leading system message on the caller's request stays at position 0; prior turns are spliced in AFTER, so history doesn't override session instructions.
- **Tool-call turns persisted** — an assistant response with `toolCalls[]` is stored with those toolCalls attached, so the next turn's context includes the tool-call trace.
- **Fail-open** — a broken store never takes the request path down. Store errors are captured via `onError` and the middleware falls through without prepending history.
- **Compose with `compactHistory`** (v1.91.0) as the summarizer when using `pruneStrategy: 'summarize'` — LLM-summarized context compression without hand-rolled logic.
- MCP resource: `config://session-context-store` (`hitRate`, turns-appended / prunes / summarizations / store-errors).

Composition rule: put `sessionContextStore` **outside** `guardrails` / `promptInjectionGuard` (those need to see the merged history — prior turns can contain injected content too) and **outside** `semanticCache` / `responseCache` (a stateful chat is context-dependent and shouldn't share cache entries with other sessions).

## Function-call arbitrator (new in v2.18.0)

`functionCallArbitrator` is a policy + validation layer for LLM tool calls. Each shipped provider already normalizes tool schemas to `{ name, description?, input_schema }` on outbound and tool-call responses to `{ id, name, input }` on inbound. This middleware sits **on top** to enforce cross-provider consistency + safety.

```js
const { functionCallArbitrator } = require('@saptarishi/cds-plugin-llm');

llm.use(functionCallArbitrator({
  tools: [
    {
      name: 'lookup_customer',
      description: 'Look up a customer by ID',
      input_schema: {
        type: 'object',
        required: ['customerId'],
        properties: { customerId: { type: 'string' } },
      },
    },
    // ... more tools
  ],
  onInvalid:         'strip',       // 'throw' | 'strip' | 'log'
  allowUnregistered:  false,
  onCall:        (i) => cds.log('llm:tools').info('called', i.call.name),
  onInvalidCall: (i) => cds.log('llm:tools').warn('rejected', i),
}));
```

What it enforces:
- **Registered-tool allowlist** — tool calls to names not in the registry are rejected by default. Prevents the model from calling tools you never intended to expose.
- **Per-call JSON Schema validation** — every `result.toolCalls[].input` is validated against the tool's `input_schema` using the shipped `validateBuiltIn` from `structuredOutputValidator`. Catches missing required fields, wrong types, unexpected props.
- **Three-way policy** on invalid calls:
  - `'throw'` → raise `InvalidToolCallError` (`code: 'INVALID_TOOL_CALL'`) with `.callId`, `.name`, `.errors[]`
  - `'strip'` → silently drop the invalid call from `result.toolCalls` (default — fail-soft)
  - `'log'` → keep the call but tag it with `.invalid = true` + `.invalidErrors[]` for downstream inspection

**Cross-provider tool shape normalization** on outbound (opt-out via `normalizeOutbound: false`) accepts Anthropic (`input_schema`), OpenAI (`parameters`), aliases (`schema`), and Gemini (`functionDeclarations` wrapper) forms — all normalized to the canonical `input_schema` shape before the provider sees the request.

Standalone helpers `normalizeToolShape(tool)` and `normalizeToolList(tools)` are exported for one-off use outside the middleware chain (offline shape migration, tests, config validation).

MCP resource: `config://function-call-arbitrator` (registered tool names + policy + stats + `invalidRate()`).

Placement rule: put `functionCallArbitrator` **inside** prompt-safety middleware (`guardrails`, `promptInjectionGuard`) so those still guard the raw text; **outside** the tool runner (`runTools`, `streamTools`, `Agent`) so the runner only ever sees validated + allowlisted calls.

## Provider load balancer (new in v2.17.0)

`providerLoadBalancer` rotates across N credential sets of the **same** provider kind — multiple OpenAI accounts to work around per-account rate limits, multiple Azure regions for geographic spread, multiple Bedrock cross-account roles, etc. Complements the shipped 11-provider abstraction (which supports one credential set per provider kind) with per-account rotation on top.

```js
const { providerLoadBalancer } = require('@saptarishi/cds-plugin-llm');

llm.use(providerLoadBalancer({
  credentials: [
    { name: 'openai-org-a', apiKey: process.env.OPENAI_KEY_A, weight: 3 },
    { name: 'openai-org-b', apiKey: process.env.OPENAI_KEY_B, weight: 1 },
    { name: 'openai-org-c', apiKey: process.env.OPENAI_KEY_C, weight: 1 },
  ],
  strategy: 'least-loaded',                                       // round-robin | least-loaded | weighted-random | sticky
  applyCredential: (req, cred) => ({ ...req, credentials: { apiKey: cred.apiKey } }),
  unhealthyThreshold:  3,                                         // 3 consecutive failures → mark unhealthy
  unhealthyCooldownMs: 30_000,
  onHealthChange: (i) => cds.log('llm:lb').warn('health', i),
}));
```

Four strategies:
- **`round-robin`** — cycle through credentials in order (default)
- **`least-loaded`** — pick the credential with the fewest in-flight calls
- **`weighted-random`** — probability ∝ `weight` (higher weight = more picks)
- **`sticky`** — same `stickyKeyOf(ctx)` value always picks the same credential (session affinity, tenant isolation across accounts)

Health tracking is **opt-in** via `unhealthyThreshold`. After N consecutive failures, a credential is marked unhealthy and skipped by the strategy until `unhealthyCooldownMs` elapses. A successful call resets the consecutive-error counter. Composes with `providerHealthProbe` (v1.62.0) — that primitive tests liveness *proactively*; this one *reacts* to real failures.

- `snapshotCredentials()` returns per-credential state (weight, inFlight, totalPicks, totalErrors, healthy, unhealthySince) for real-time SaaS dashboards.
- `markUnhealthy(name)` / `markHealthy(name)` for ops overrides.
- `AllCredentialsUnhealthyError` (code `ALL_CREDENTIALS_UNHEALTHY`) when the strategy has no healthy credential to pick — fail loud rather than silently proceed with no credential applied.
- MCP resource: `config://provider-load-balancer`.

Placement rule: put `providerLoadBalancer` **outside** providers, **inside** routing (`semanticRouter` v2.16.0, `costAwareRouter` v2.10.0). The routing decides which *model* to use; this middleware decides which *credential* to use.

## Semantic router (new in v2.16.0)

`semanticRouter` picks the model + system prompt + params by **embedding the user's request** and matching it against per-route centroids (cosine similarity). Each route is defined by a bucket of example prompts; the middleware averages their embeddings to form a centroid, then classifies incoming queries by nearest neighbor.

```js
const { semanticRouter } = require('@saptarishi/cds-plugin-llm');

llm.use(semanticRouter({
  embedder: async (text) => (await llm.embed({ input: text })).embeddings[0],
  routes: [
    {
      name:     'code',
      model:    'anthropic/claude-opus-4-7',
      system:   'You are an expert engineer. Return runnable code.',
      examples: ['Write a Python function to reverse a string', 'Debug this TS error', 'Refactor this Go handler'],
    },
    {
      name:     'procurement',
      model:    'openai/gpt-4o',
      system:   'You are an SAP procurement analyst.',
      examples: ['Analyze this vendor quote', 'Draft a PO for 500 units', 'What are the payment terms?'],
    },
    {
      name:        'chit-chat',
      model:       'openai/gpt-4o-mini',
      temperature: 0.9,
      examples:    ['Hi', 'How are you?', 'Tell me a joke'],
    },
  ],
  threshold:    0.75,
  defaultRoute: 'chit-chat',
  onRoute: (i) => cds.log('llm:router').info('routed', i),
}));
```

Complements the two other routing primitives:
- **`modelRouter`** (1.x) — static keyword/predicate routing. Deterministic, easy to reason about, but requires you to enumerate rules.
- **`costAwareRouter`** (v2.10.0) — *reactive* escalation: cheap tier first, escalate on low score.
- **`semanticRouter`** (v2.16.0) — *predictive* first pick: classify by meaning, get the right tier from the start.

Compose them: semantic router picks the tier upfront; cost-aware router escalates when the prediction was wrong.

Highlights:
- **Lazy centroids** — construction is zero-latency. First call computes example embeddings and caches them. `warmup()` pre-computes eagerly if you want deterministic startup.
- **Precomputed centroids** — ship `centroid: [1, 0, 0, ...]` on a route to skip embedding entirely. Useful for offline-computed centroids that stay stable across deploys.
- **Fail-open** — embedder or extractKey errors fall through to `next()` unmodified. The router never takes the request path down.
- **Threshold + fallback** — below-threshold queries use `defaultRoute` (or passthrough if null). Prevents forcing a bad route onto ambiguous queries.
- MCP resource `config://semantic-router` exposes route table (names, models, example counts, centroid readiness), threshold, default route, and route distribution.

## Retry budget (new in v2.15.0)

`retryBudget` is a Google-SRE-style global cap on the retries-to-requests ratio in a rolling window. It prevents **retry storms** during partial outages: with `retryOnRateLimit` + `regionFailover` + `speculativeHedge` + `costAwareRouter`'s escalate-on-error combined, one call can trigger up to N×M×K upstream attempts. If 10k concurrent requests each retry 5x, you fire 50k requests at an already-failing upstream — a self-inflicted DoS. The retry budget refuses further retries when the ratio breach is real.

```js
const { retryBudget, retryOnRateLimit } = require('@saptarishi/cds-plugin-llm');

const budget = retryBudget({
  retryRatio:    0.10,     // SRE-canonical: max 10% retries
  windowMs:      60_000,
  minSampleSize: 100,      // don't trip on cold-start noise
  onExhausted: (i) => cds.log('llm:budget').warn('exhausted', i),
  onLowBudget: (i) => cds.log('llm:budget').info('near-cap', i),
});

llm.use(retryOnRateLimit());   // outer — retries re-enter budget
llm.use(budget);               // inner — sees each retry
```

How retries are detected: a `WeakMap` tracks per-ctx call counts. The first pass for a given ctx is a request; every subsequent pass with the **same ctx reference** is a retry. Works because retry primitives (`retryOnRateLimit`, `autoRetry`, `regionFailover`, `costAwareRouter`'s escalation) re-invoke `next()` with the same ctx. Middlewares that create fresh ctx per attempt (`speculativeHedge`) look like separate requests — correct, because each hedge IS a separate upstream call.

Highlights:
- **SRE-canonical default** of 10% retries matches the Google SRE book.
- **`minSampleSize` gate** (default 100) prevents tripping on cold-start noise. Ratio check is skipped until enough requests accumulate.
- **Rising-edge callbacks** at 50% and 80% of the retry budget (configurable). Level is only re-fired after ratio drops below the lowest threshold — no spam.
- **`RetryBudgetExhaustedError`** (code `RETRY_BUDGET_EXHAUSTED`) with `.currentRatio`, `.retryRatio`, `.requests`, `.retries`, `.windowMs`.
- MCP resource `config://retry-budget` exposes live counts, current ratio, budget fraction, per-threshold fire counts.

Composition rule: place `retryBudget` **inside** the retry primitives (outer: retry; inner: budget) so retries re-enter the budget — that's how it observes them. Test-drive the whole stack with `chaosInjector` (v2.11.0) to verify budget behavior under simulated outages before hitting prod.

## Multi-tenant fair-share scheduler (new in v2.14.0)

`fairShareScheduler` sits above the shipped concurrency primitives and admits requests in weighted round-robin (WRR) order across tenants. No single tenant can starve the others under load — solves the noisy-neighbor problem in SaaS deployments of the plugin.

```js
const { fairShareScheduler } = require('@saptarishi/cds-plugin-llm');

llm.use(fairShareScheduler({
  tenantOf:          (ctx) => ctx.request.tenantId ?? 'anon',
  maxConcurrent:     20,
  weights:           { gold: 5, silver: 2, free: 1 },
  defaultWeight:     1,
  maxPerTenantQueue: 100,
  onReject: (i) => cds.log('llm:fair-share').warn('backpressure', i),
}));
```

Distinct from the shipped primitives:
- **`tenantIsolate`** just TAGS requests for observability. `fairShareScheduler` *schedules*.
- **`bulkhead`** caps concurrency globally without fairness. `fairShareScheduler` cap + share fairly.
- **`distributedLock`** is per-key exclusivity across instances. `fairShareScheduler` is per-tenant queueing within an instance.

Highlights:
- **WRR**: `weights: { gold: 3, silver: 1 }` gives gold 3× more slots per cycle. When all tenants with pending work run out of credit, credits refill from weight — but only for tenants with pending work, so idle tenants don't accumulate unfair credit debt.
- **Backpressure**: per-tenant queue depth cap (`maxPerTenantQueue`). Exceeding throws `FairShareRejectedError` (code `FAIR_SHARE_QUEUE_FULL`) with `.tenant`/`.queueDepth`/`.queueLimit`. One over-subscribed tenant cannot block others — separate per-tenant queues.
- **Live snapshot**: `snapshotTenants()` returns per-tenant weight, credits, active, queued, admitted, rejected. Great for real-time SaaS dashboards. MCP resource: `config://fair-share-scheduler`.

Composition rule: place `fairShareScheduler` **outside** the shipped `bulkhead` (the scheduler manages its own ceiling; wrapping bulkhead underneath would double-count slots). Place `retryOnRateLimit` **outside** the scheduler so per-tenant retries consume per-tenant queue slots (respect fairness); placing it **inside** means retries bypass fairness — appropriate only for short internal loops.

## Reversible PII tokenization (new in v2.13.0)

`reversibleTokenization` strips PII from the outbound request (replaces with opaque `<TYPE_N>` tokens like `<EMAIL_1>`, `<SSN_1>`), sends the tokenized prompt to the model, then restores the original values in the response before returning. The **model never sees raw customer data** but callers get useful answers back. Ideal for GDPR / HIPAA / financial compliance workloads where sending PII to an external LLM is legally or contractually restricted.

```js
const { reversibleTokenization } = require('@saptarishi/cds-plugin-llm');

llm.use(reversibleTokenization({
  patterns: {                              // extends the built-in set
    BADGE_ID: /\bE\d{6}\b/g,
  },
  onTokenize: (i) => cds.log('llm:pii').info('scrubbed', i.byType),
  onUnknownToken: () => '[REDACTED]',     // rewrite hallucinated tokens
}));

// Then:
const r = await llm.chat({ prompt: 'Send invoice to alice@x.com about SSN 123-45-6789.' });
// Model saw:  'Send invoice to <EMAIL_1> about SSN <SSN_1>.'
// Model said: 'I sent the invoice to <EMAIL_1> regarding <SSN_1>.'
// Caller gets:'I sent the invoice to alice@x.com regarding 123-45-6789.'
```

Built-in patterns (low false-positive taxonomy): `EMAIL`, `CREDIT_CARD`, `SSN`, `PHONE`, `IPV4`, `IBAN`. `PERSON` / `ADDRESS` are intentionally NOT included — regex-based name detection is too noisy; compose with a proper NER model instead.

Highlights:
- **Dedup per request** — same value → same token everywhere. `alice@x.com` in both `system` and `messages[]` becomes the same `<EMAIL_1>`. Reduces token cost and makes downstream detokenization exact.
- **Hallucinated tokens** — if the model returns `<EMAIL_99>` (no such token was ever emitted), it's left as-is by default; override via `onUnknownToken(token) => replacement` to redact.
- **Standalone helpers** — `tokenizePII(text)` and `detokenizePII(text, mapping)` exported for one-off scrubbing (offline scripts, audit-log preparation, unit tests of pattern sets).
- MCP resource `config://reversible-tokenization` exposes pattern types, `restorationRate`, tokens-created / restored / unknown, per-type counts.

Composition rule: place `reversibleTokenization` **inside** `guardrails` / `promptInjectionGuard` (those need raw text) and **outside** any cache (`semanticCache`, `responseCache`) so tokenized prompts become the cache keys and PII never leaks into shared caches.

## Speculative hedging (new in v2.12.0)

`speculativeHedge` fires the same request to N candidates with staggered delays; the first successful reply wins, losers are signalled to abort. Trades $ for tail latency — pay for a small amount of duplicate work to guarantee p99 when your SLO is p99 (not p50).

```js
const { speculativeHedge } = require('@saptarishi/cds-plugin-llm');

llm.use(speculativeHedge({
  candidates: [
    { name: 'primary' /* fires immediately */ },
    { name: 'us-east', hedgeDelayMs: 200, modifyRequest: (r) => ({ ...r, model: 'us-east/gpt-4o' }) },
    { name: 'eu-west', hedgeDelayMs: 500, modifyRequest: (r) => ({ ...r, model: 'eu-west/gpt-4o' }) },
  ],
  isSuccess: (r) => !!r?.text,
  onWin:  (i) => cds.log('llm:hedge').info('win', i),
  onLoss: (i) => cds.log('llm:hedge').info('lost', i),
}));
```

Highlights:
- **Per-candidate config** — each entry can carry its own `hedgeDelayMs` + `modifyRequest`. Common patterns: swap `model`, route to different regions, or hit different provider endpoints.
- **Loser cancellation** via Web-standard `AbortSignal` on `ctx.signal`. Providers that respect it early-exit; providers that don't keep running but their result is discarded (token cost may still be billed — choose `hedgeDelayMs` conservatively).
- **`isSuccess(result)`** — treat a returned result as a losing hedge (still racing). Useful for "keep waiting for a better one" against low-confidence responses.
- **Cost visibility** — `hedgeRatio()` = `hedgesLaunched / totalCalls`; per-candidate `winsByCandidate` + `launchesByCandidate` counters identify the reliable candidate. MCP resource: `config://speculative-hedge`.
- **`AllHedgesFailedError`** (`code: 'ALL_HEDGES_FAILED'`) — thrown when every hedge errors or returns non-success. Carries `.errors[]` and `.candidateNames[]`.

Placement rule: put `speculativeHedge` **outside** any per-attempt retry / bulkhead / breaker. Each hedge should consume its own retry / concurrency budget — otherwise a shared retry counter is drained by hedges the winner is racing against.

## Chaos injector (new in v2.11.0 — test-only)

`chaosInjector` gives you a way to verify that the shipped resilience stack — `retryOnRateLimit`, `circuitBreaker`, `bulkhead`, `adaptiveRateLimit`, `requestCoalescer`, `costAwareRouter` — actually behaves correctly when the provider misbehaves. Given a seed, it reproduces the exact same fault sequence, so a red CI run can be re-run locally byte-identical.

```js
const { chaosInjector, retryOnRateLimit, circuitBreaker } = require('@saptarishi/cds-plugin-llm');

llm.use(retryOnRateLimit());
llm.use(circuitBreaker());
// Injector sits INNERMOST — closest to the provider — so all outer
// resilience middleware sees the fault and can react.
llm.use(chaosInjector({
  seed: 42,
  faults: {
    rate429:     0.30,
    rateTimeout: 0.10,
    rateGarbage: 0.05,
  },
}));
```

Fault types (stable priority order — `networkError → timeout → 500 → 503 → 429 → garbage → slow`; first roll that hits wins):
- **`rateNetworkError`** → `ECONNRESET`
- **`rateTimeout`** → `ETIMEDOUT`
- **`rate500` / `rate503` / `rate429`** → HTTP status errors with `.status`/`.statusCode` set
- **`rateGarbage`** → calls `next()` but replaces `result.text` with malformed content (empty string, truncated JSON, HTML error pages) so downstream parsers/validators see a real broken response
- **`rateSlow`** → adds a configurable delay before calling `next()`

**Safety guard**: `chaosInjector` refuses to construct unless one of `NODE_ENV=test`, `CHAOS_INJECTOR_ENABLED=1`, or `iKnowThisIsChaos: true` is present. That way an accidental prod deploy of a fault-injecting middleware throws loudly at boot instead of silently degrading requests. MCP resource: `config://chaos-injector` (exposes seed + rates + `lastFault` + `lastSeedRoll` for debugging non-reproducible failures).

## Cost-aware model router (new in v2.10.0)

`costAwareRouter` tries a **cheap-tier** model first; if the response scores below your quality threshold, it escalates to the **premium tier**. Ties together three previously-shipped primitives — `modelRouter` (static routing), `scoreResponse` (mechanical scoring), and `costForecast` (budgeting) — into an economic win pattern: most calls run cheap; only the hard ones escalate.

```js
const { costAwareRouter, scoreResponse } = require('@saptarishi/cds-plugin-llm');

llm.use(costAwareRouter({
  tiers: [
    { model: 'gpt-4o-mini', pricePerMtokIn: 0.15, pricePerMtokOut: 0.60 },
    { model: 'gpt-4o',      pricePerMtokIn: 2.50, pricePerMtokOut: 10.00 },
  ],
  scorer:         (result) => scoreResponse(result, { rubric: myRubric }).score,
  scoreThreshold: 0.75,
  onEscalate: (i) => cds.log('llm:router').info('escalate', i),
  onFinal:    (i) => cds.log('llm:router').info('final',    i),
}));
```

Highlights:
- **Multi-tier**: 2, 3, or N tiers. `maxEscalations` caps how far the router will escalate.
- **`escalateOnError`** (default true): a downstream throw on the cheap tier promotes to the next tier instead of surfacing the error — this is a *quality-driven* re-attempt on a *different model*, distinct from `retry` (transport-error re-attempt on the *same* one).
- **Fail-safe scorer**: exceptions or non-numeric returns count as failing scores. Safer to escalate than to fail loudly.
- **Cost accounting**: when tier pricing is set, `tokensSpentUsd` / `tokensSavedUsd` / `savingsRatio()` are computed from the shipped `result.usage`. Live-visible via `config://cost-aware-router`.

Placement rule: put `costAwareRouter` **outside** any middleware that reads `request.model` (retry, bulkhead, providers). The router mutates `request.model` per attempt so downstream sees each tier as a fresh call.

## Structured-output auto-repair (new in v2.9.0)

`structuredOutputRepair` upgrades the "response failed schema validation" story from **fail** (v1.x `structuredOutputValidator` in `throw` mode) or **fail-then-re-ask** (validator with `onInvalid: 'retry'`) to a **strategy chain**: cheap deterministic local fixes are tried first, and the LLM is only re-asked when local repair fails. Most fixable outputs cost zero extra tokens.

```js
const { structuredOutputRepair, schemas } = require('@saptarishi/cds-plugin-llm');

llm.use(structuredOutputRepair({
  schemaFrom: (ctx) => ctx.request.format,
  strategies: ['json-fix', 're-ask'],   // order matters
  maxLlmRetries: 1,
  onRepair: (info) => cds.log('llm:repair').info(info),
}));
```

Built-in strategies:
- **`json-fix`** — strips ```json fences, narrows to the outer `{...}` block, removes `//` and `/* */` comments (respecting quoted regions), removes trailing commas, quotes unquoted keys, converts single-quoted strings to double-quoted. Deterministic, idempotent for already-valid JSON.
- **`re-ask`** — sends a corrective user message with the schema violations inline, bounded by `maxLlmRetries`. Retry-result is propagated so callers see the corrected text/data, not just the parsed object.

Custom strategies plug in as `{ name, apply(rawText, { errors, schema, parsed }) }`. Throwing custom strategies are skipped silently and the chain continues. All strategies share the same `StructuredOutputInvalidError` on exhaustion → unified downstream error handling with the 1.x validator. MCP resource: `config://structured-output-repair` (strategy list + per-strategy hit counts + LLM retries used).

Standalone `jsonAutoFix(text)` is also exported for one-off cleanup outside a middleware chain (useful in offline scripts).

## Request coalescer (new in v2.8.0)

`requestCoalescer` is the in-flight sibling of `semanticCache`. When N concurrent requests arrive with the same key (same prompt + model + system by default), only ONE upstream call fires — all N callers await the same shared Promise and receive the same result. Fixes the classic **cache-stampede** on cold caches: the first miss no longer triggers N parallel misses racing to populate.

```js
const { requestCoalescer, semanticCache } = require('@saptarishi/cds-plugin-llm');

llm.use(requestCoalescer({
  ttlMs: 500,               // also coalesce briefly-following requests
  onCoalesce: (i) => cds.log('llm:coalesce').info(i),
}));
llm.use(semanticCache({ embedder, store }));
```

- **Streaming methods are skipped** by default (a consumed stream can't be fanned out). Overridable via `skipMethods`.
- **`cloneResult: true`** deep-clones the shared result per caller if downstream code mutates it.
- **`keyPrefix`** namespaces coalescing across tenants; **`maxInFlightKeys`** caps the map size as a safety valve during pathological fan-out.
- MCP resource `config://request-coalescer` exposes `savingsRatio`, in-flight + recently-settled counts, `peakInFlight`, and lead/coalesce/ttl-hit counters.

Composition rule: wrap `requestCoalescer` **outside** `semanticCache`/`responseCache` to absorb the burst before it hits the cache, and **outside** `bulkhead`/`retry` — one shared upstream call should only consume one slot and one retry budget, not N.

## Adaptive rate-limit tuner (new in v2.6.0)

`adaptiveRateLimit` watches the rate-limit headers the provider sends back on every call (parsed by the shipped OpenAI / Anthropic / Gemini / Bedrock providers into `result._rateLimit`) and dynamically adjusts the paired `bulkhead()`'s `maxConcurrent` to stay within remaining quota. On a 429 or 503 it immediately halves concurrency; on a healthy sample it grows back toward `(remainingRatio − headroom) × maxConcurrent`.

```js
const { bulkhead, adaptiveRateLimit } = require('@saptarishi/cds-plugin-llm');

const bh = bulkhead({ maxConcurrent: 20 });
llm.use(bh);
llm.use(adaptiveRateLimit({
  bulkhead:      bh,
  headroom:      0.20,       // stay 20% below remaining quota
  minConcurrent: 2,
  maxConcurrent: 50,
  onAdjust: (i) => cds.log('llm:arl').info(i),
  on429:    (i) => cds.log('llm:arl').warn('429', i),
}));
```

Composes with `adaptiveBulkhead` (1.61) — that one is **latency-driven** (AIMD on p95), this one is **quota-driven** (headroom against remaining requests/tokens). Both can call `setMaxConcurrent` on the same bulkhead; they naturally settle on the tighter of the two signals. MCP resource: `config://adaptive-rate-limit`.

## Response scoring + consensus voting (new in v2.4.0 · v2.5.0)

Two composable primitives for output quality:

- **`scoreResponse`** (v2.4.0) — checks-based grader with a frozen `CHECKS` catalog (`no-hallucinated-numbers`, `min-length`, `contains`, `matches`, `no-refusal`, ...). Returns a score in `[0, 1]` plus per-check details. Wire it as a soft signal for logging + dashboards, or as a hard gate via `shouldCache: (_c, r) => scoreResponse(r).score >= 0.8` in `semanticCache`.
- **`consensusVoting`** (v2.5.0) — fan out to N models in parallel; return the majority winner with configurable quorum, comparators (`equalityIgnoreCase`, `jsonDeepEqual`, `numeric`, `firstToken`, ...), and confidence. Useful for compliance-sensitive answers where a single-model reply is not defensible.

## Cost forecasting (new in v2.1.0)

`costForecast` projects your rolling burn rate against a monthly budget, raising `warn` and `critical` events before you hit the wall (unlike `costBudget`, which fires when spend has already crossed a threshold). Straight-to-critical transitions correctly fire the `warn` callback first. MCP resource: `config://cost-forecast`.

## Provider capability probe (new in v2.3.0)

`capabilities(llm)` returns a live capability report for the configured provider + model — what actually works, not what a hand-maintained matrix claims. Complements the static [Provider capability matrix](#provider-capability-matrix) below with runtime evidence: what tools work, whether structured outputs land, whether the model actually streams, prompt-cache support, and vision / PDF handling.

```js
const { capabilities } = require('@saptarishi/cds-plugin-llm');

const report = await capabilities(llm, { live: false });
// {
//   provider: 'llm-anthropic', model: 'claude-sonnet-4-6',
//   chat: true, stream: true, tools: true, structuredOutputs: true,
//   embed: false, promptCache: true, vision: true, pdf: true, ...
// }
```

Pass `{ live: true }` to actually round-trip a probe request per capability (~5s) — useful for CI verification against a real endpoint. `PROVIDER_CAPABILITY_MATRIX` + `MODEL_CAPABILITY_OVERRIDES` exports let you consume the static matrix directly for gating in code.

## Full API

```ts
llm.chat({
  messages: [{ role: 'user' | 'assistant' | 'tool', content: string | ContentBlock[], toolCalls?, tool_call_id? }],
  system?: string,
  model?: string,          // overrides configured default
  maxTokens?: number,      // default 16000 (lower for Groq free tier — TPM: 12k)
  format?: JSONSchema,     // unified structured output; returns parsed data
  tools?: Tool[],          // [{ name, description, input_schema }]
  thinking?: { type: 'adaptive' } | false,  // Anthropic-only; default adaptive
  cache?: boolean,         // Anthropic-only; caches the system prompt
  retries?: { max, baseMs, maxMs },
}) => Promise<{
  text: string,
  data?: unknown,          // populated when format was set
  toolCalls?: [{ id, name, input }],  // populated when model called tools
  raw: unknown,            // provider-native response
  usage: { input_tokens, output_tokens, ... },
  stopReason: string,      // 'end_turn' | 'tool_use' | 'max_tokens' | ...
  model: string,
}>

llm.embed({ input: string | string[], model? })
  => Promise<{ embeddings: number[][], model: string }>
```

## Provider capability matrix

|                       | `llm-anthropic` | `llm-ollama` | `llm-groq` | `llm-openai-compatible` | `llm-genai-hub` |
|---|---|---|---|---|---|
| chat                  | ✓ | ✓ | ✓ | ✓ | ✓ |
| stream                | ✓ (SDK)        | ✓ (NDJSON)   | ✓ (SSE)   | ✓ (SSE)                 | ✓ (SSE) |
| structured output (`format`) | ✓ (`output_config`) | ✓ (native `format`) | ✓ (json_object mode) | ✓ (json_object) | ✓ (json_object) |
| tool use (`tools`)    | ✓ (native)     | ✓ (qwen2.5, llama3.1+) | ✓ (function-calling models) | ✓ | ✓ |
| vision (images)       | ✓ (Claude 3.5+) | ✓ (llava, moondream, llama3.2-vision) | ✓ (llama-4-scout, etc.) | ✓ (gpt-4o, etc.) | ✓ (deployment-dependent) |
| PDF (documents)       | ✓ (Claude 3.5+ native)  | — (render pages to images first) | — (Groq doesn't accept files) | ✓ (base64, on models that support `file` blocks — GPT-4o+) | ✓ (deployment-dependent) |
| embeddings            | — (no first-party embeddings) | ✓ (`mxbai-embed-large`, etc.) | ✓ (when model available) | ✓ (`text-embedding-3-*`, `ada-002`, etc.) | ✓ (needs `embeddingDeploymentId`) |
| response cache (opt-in) | ✓ | ✓ | ✓ | ✓ | ✓ |
| prompt caching (`cache`) | ✓ (Anthropic system prompt ephemeral) | — | — | — | — |
| adaptive thinking (`thinking`) | ✓ (Opus 4.7 native) | — | — | — | — |

## FAQ

**How does this relate to `@cap-js/ai`?**
`@cap-js/ai` is scoped to value-help recommendations and SAP-RPT-1 with SAP AI Core integration. This plugin is a general-purpose LLM client with multi-provider support and a broader feature surface (streaming, tool use, vision, structured output). The two can coexist: one CAP app can `cds.connect.to('ai')` for value-help features and `cds.connect.to('llm')` for direct LLM calls.

**Can I use this without SAP BTP?**
Yes. Only the `llm-genai-hub` kind requires BTP (specifically AI Core). The other four kinds (Anthropic, Ollama, Groq, OpenAI-compatible) work in any Node.js environment. This is deliberate — the plugin is useful for CAP apps that don't run on BTP, and useful for prototyping before a BTP deployment.

**Is this production-ready?**
As of `1.0.0`: the public API surface is committed under semver (breaking changes require a major bump). The core is functional and unit-tested (64 tests + wire-protocol E2E). The plugin is deployed live on SAP BTP Cloud Foundry through this repo's `joule-project-api`. The one remaining honesty note: the GenAI Hub provider is spec-compliant and mock-verified end-to-end but not yet live-verified against a real AI Core `extended` deployment — happy to accept a PR from anyone with access. Everything else has been live-verified against its target service.

**How do I add a new provider?**
Extend `LLMService` (or `OpenAICompatibleLLMService` if the target speaks the OpenAI shape), implement `_chat` (required), plus `_stream` and `_embed` if applicable. Register a kind in your `package.json` under `cds.requires.kinds.<my-provider>` with `impl` pointing at the new class file and `external: true`.

**Which model do you recommend for common tasks?**
- Structured extraction / classification: any 7B+ instruction-tuned model. Groq's `llama-3.3-70b-versatile` is a good default (fast + free tier).
- Vision (invoice OCR, chart reading): Claude Opus 4.7 or GPT-4o for accuracy; Groq's `meta-llama/llama-4-scout-17b-16e-instruct` or Ollama's `llava` for local/cheap.
- Long-context summarization: Claude Opus 4.7 (1M context) or a GenAI Hub deployment of the same.
- Tool use / agentic loops: Claude 3.5+ or qwen2.5 on Ollama for multi-step reliability. Groq's llama models work for single-tool cases.
- Embeddings: Ollama with `mxbai-embed-large` or `nomic-embed-text`.

**Why not just use `@anthropic-ai/sdk` or `openai` directly?**
Three reasons: (1) CAP idiom — `cds.connect.to('llm')` is more natural in a CAP handler than importing an SDK class. (2) Provider swap without code change — flip a config value from `llm-groq` to `llm-anthropic` and the same handler works. (3) Unified interface for tools + structured output + streaming across all providers, so you don't rewrite the message-translation code five times.

**How do I do RAG (retrieval-augmented generation)?**
The plugin gives you `embed()` — combine with a vector store to complete the loop. Companion package [`@saptarishi/cds-plugin-vector-hana`](https://www.npmjs.com/package/@saptarishi/cds-plugin-vector-hana) provides a SAP HANA Cloud vector store (native `REAL_VECTOR` + `COSINE_SIMILARITY`) with a SQLite fallback for local dev, so you can build semantic-search features without HANA access and swap backends at deploy time.

**What happens if the underlying provider's API changes?**
Each provider adapter is a thin file (~150 lines). Provider API changes are localized to one file. The plugin's public surface (`chat`, `stream`, `embed`) is stable across provider changes.

## Contributing

PRs and issues welcome. The [repo](https://github.com/kalyanjanumpally/sap-joule-ai-plugin) has the plugin as `cds-plugin-llm/`. Standard workflow:

```sh
git clone https://github.com/kalyanjanumpally/sap-joule-ai-plugin
cd sap-joule-ai-plugin/cds-plugin-llm
npm install
npm test              # 39 unit tests, no external deps
npm run typecheck     # TypeScript definition check
node ../scripts/verify-genai-hub.js   # E2E mock verification
```

CI runs the same checks on every push (Node 20 + 22 matrix).

**Highest-value contributions right now:**
- Live-verification of the GenAI Hub provider against a real AI Core `extended` deployment
- Embeddings support for OpenAI-compatible providers
- Additional structured-output modes (JSON schema strict on models that support it)

## Roadmap

- ~~**0.7**: embeddings on OpenAI-compat / Groq~~ ✓ shipped in v0.7.0
- ~~**0.8**: PDF content blocks (Anthropic native; other providers explicit-reject)~~ ✓ shipped in v0.8.0
- ~~**0.9**: OpenAI-compat inline PDF (via `file` content-block) + GenAI Hub embeddings + response caching layer~~ ✓ shipped in v0.9.0
- ~~**1.0**: API stability commitment~~ ✓ shipped in v1.0.0 (live verification of GenAI Hub open — see FAQ)
- ~~**1.1**: `runTools()` — automatic multi-turn tool-use loop~~ ✓ shipped in v1.1.0
- ~~**1.2**: middleware / interceptor pattern~~ ✓ shipped in v1.2.0
- ~~**1.3**: built-in `rateLimit` + `otel` middlewares; vector store `upsertMany`~~ ✓ shipped in v1.3.0 (llm) / v0.2.0 (vector-hana)
- ~~**1.4**: `redisRateLimit` middleware; HANA HNSW index config~~ ✓ shipped in v1.4.0 (llm) / v0.3.0 (vector-hana)
- ~~**1.5**: `saptarishi-llm` CLI~~ ✓ shipped in v1.5.0
- ~~**1.6**: `saptarishi-llm init` CAP-app scaffolder~~ ✓ shipped in v1.6.0
- ~~**1.7**: `saptarishi-llm mcp` server (MCP over stdio)~~ ✓ shipped in v1.7.0
- ~~**1.8**: `PromptRegistry` + MCP resources + MCP prompts~~ ✓ shipped in v1.8.0
- ~~**1.9**: `loadFromDir` prompt loader + `--prompts-dir` on MCP + scaffold SSE endpoint + MCP resource templates~~ ✓ shipped in v1.9.0
- ~~**1.10**: MCP HTTP+SSE transport (`--http`) + hot-reload prompts (`--watch-prompts`)~~ ✓ shipped in v1.10.0
- ~~**1.11**: MCP bearer-token auth (`--auth-token`)~~ ✓ shipped in v1.11.0
- ~~**1.12**: MCP progress notifications on `tools/call`~~ ✓ shipped in v1.12.0
- ~~**1.13**: MCP list-changed notifications (broadcasts on hot-reload)~~ ✓ shipped in v1.13.0
- ~~**1.14**: OpenAI Files API for URL PDFs (`uploadPdfFromUrl`)~~ ✓ shipped in v1.14.0
- ~~**1.15**: Azure OpenAI provider~~ ✓ shipped in v1.15.0
- ~~**1.16**: MCP OAuth2 / JWKS-based JWT auth + pluggable `authTokenVerifier`~~ ✓ shipped in v1.16.0
- ~~**1.17**: MCP resource subscriptions (`resources/subscribe` + `notifications/resources/updated`)~~ ✓ shipped in v1.17.0
- ~~**1.18**: per-session provider overrides (`--providers-config`, `initialize._meta.provider`, per-call `provider` arg)~~ ✓ shipped in v1.18.0
- **1.19+**: CAP model registry (providers in `.cds` files), Bedrock + Vertex provider paths, mTLS auth for MCP HTTP, Anthropic prompt caching
- **Companion package**: [`@saptarishi/cds-plugin-vector-hana`](https://www.npmjs.com/package/@saptarishi/cds-plugin-vector-hana) — HANA Cloud vector store + SQLite fallback for RAG

## License

Apache-2.0