# Architecture Specification — Project 4: Real-Time Low-Latency Web Chatbot

> **Status:** Draft | **Version:** 1.0.0 | **Scope:** Project 4 — FastAPI (ASGI) + React SSE

---

## 0. Methodological Note

The standard template (synchronous backend / request-response) does not natively cover three core dimensions of Project 4: **token-by-token streaming**, **multi-tenant concurrency isolation**, and the **frontend contract**. I have preserved the structure and numbering of the original template identically (Sections 1 to 6) while adapting them to the SSE flow, and added three sections (7, 8, 9) absent from the original model. Each addition is justified at its point of introduction.

---

## 1. System Topology & Flow

The template establishes an `Entry → Config/Schemas → Core/Cache/RAG → Harness → Output Schema → FinOps → Output` flow. In streaming mode, this logical flow remains valid, but the `Output` is no longer a single object—it is a **sequence of events emitted while the Harness is still active**. The diagram is redrawn to emphasize the emission loop:

```text
┌─────────────────────────┐
│   ENTRY / API            │  POST /api/chat/stream (async def)
└────────────┬─────────────┘
             │
             ▼
┌─────────────────────────┐
│  Config & Schemas        │  ChatPayload (Pydantic V2, frozen)
└────────────┬─────────────┘
             │
             ▼
┌─────────────────────────┐
│  Input Guardrails        │  Prompt-injection filtering, payload size limit
└────────────┬─────────────┘
             │
             ▼
┌─────────────────────────┐
│  Harness (LLM Client)    │  Pooled async client (lifespan), stream=True
└────────────┬─────────────┘
             │  for each delta received from upstream socket
             ▼
┌─────────────────────────┐
│  StreamChunk Schema      │  PER-CHUNK Pydantic validation (not bulk)
└────────────┬─────────────┘
             │
             ▼
┌─────────────────────────┐
│  SSE Event Generator     │  event: message | event: error | event: end
└────────────┬─────────────┘
             │
             ▼
┌─────────────────────────┐
│  FinOps (post-stream)    │  Token/cost aggregation AFTER event: end
└────────────┬─────────────┘
             │
             ▼
    [ Client — ReadableStream / EventSource ]

```

**Notable deviation vs. template:** FinOps can no longer run as a single synchronous passage point—it must accumulate metrics across chunks and flush them at `event: end` (or upon cancellation). See Section 8.

---

## 2. AI Architecture Rules

> ⚠️ **STRICT COMPLIANCE:** Any diff violating these rules must be rejected.

### Rule 1 — Layered Dependency Flow

Unchanged: `Entry → Core → Clients → Schemas/Exceptions`. Modules under `schemas/chat.py` must not import any internal modules.

### Rule 2 — Exception Shielding ("Zero Naked Crash")

**Adapted.** The template assumes a clean exit with structured logs upon failure. In streaming, an exception occurring *after* the initial bytes have been sent to the client can no longer produce a standard HTTP error code (headers have already been sent). The rule becomes:

* Any exception raised before the first `yield` → standard `HTTPException` (unchanged template behavior).
* Any exception raised after the first `yield` → must be caught and transformed into `event: error\ndata: {...}\n\n`, never a raw stack trace over the socket.

### Rule 3 — Deterministic Data Contracts

Unchanged in principle (`Pydantic V2`, `frozen=True`, no raw `dict` at boundaries). **Point of vigilance:** In Project 4's reference code, `ChatMessage`, `ChatPayload`, and `StreamChunk` are not declared `frozen=True`—this is a template non-compliance that must be fixed in the final implementation (see Section 3).

### Rule 4 — FinOps Observability

Retained but expanded—see Rule 7 and Section 8, as a token-by-token stream introduces metrics not accounted for in the base template (TTFT, inter-token throughput).

### Rule 5 — Pure Logic & Side-Effect Isolation

Unchanged. The SSE generator (`stream_chat_response`) is the sole network I/O point; it lives in `services/llm.py`, never in `main.py`.

### Rule 6 — Guardrails & Security

Unchanged. User payload (message content) must pass through anti-prompt-injection filtering before reaching the Harness—absent from provided reference code, to be added in `core/guardrails.py`.

### Rule 7 — Connection Lifecycle *(added)*

Absent from the original template, which does not assume an application-scoped LLM client. Required here because the project's technical rules mandate a single pooled client initialized during `lifespan`:

* The async client (`AsyncOpenAI` / `httpx.AsyncClient`) is instantiated **once** at ASGI startup, never per request.
* Explicit cleanup (`await client.aclose()`) on `lifespan` shutdown, currently missing from reference code (where the shutdown block is empty—must be filled).

### Rule 8 — Concurrency Isolation *(added)*

Absent from the template, which targets one caller at a time. Required here as 25% of the evaluation explicitely covers event loop non-starvation:

* Zero blocking calls (`time.sleep`, synchronous HTTP client) within `async def`.
* Any CPU-bound transformation (local tokenization, heavy parsing) must run via `asyncio.to_thread`.
* A user canceling (`AbortController` client-side) must free the upstream server socket without waiting for the natural end of the stream—otherwise slow/cancelled clients will starve workers.

---

## 3. Core Contract

The template enforces `frozen=True, extra="forbid"` on all boundary models. Corrected version of Project 4 schemas (the provided reference code violates this rule—deviation noted in Rule 3):

```python
from datetime import datetime
from typing import List, Literal, Optional
from pydantic import BaseModel, ConfigDict, Field

class BaseDomainModel(BaseModel):
    model_config = ConfigDict(frozen=True, extra="forbid")

class ChatMessage(BaseDomainModel):
    role: Literal["system", "user", "assistant"]
    content: str = Field(..., min_length=1)

class ChatPayload(BaseDomainModel):
    messages: List[ChatMessage] = Field(..., min_length=1)
    model: str = Field(default="gpt-4o-mini")
    temperature: float = Field(default=0.3, ge=0.0, le=2.0)  # see Rule 3 template: <=0.3 default

class StreamChunk(BaseDomainModel):
    content: Optional[str] = None
    error: Optional[str] = None

class StreamFinOpsReport(BaseDomainModel):
    """Absent from reference code — required by Rule 4/8 (Streaming FinOps)."""
    task_id: str
    timestamp: datetime = Field(default_factory=datetime.utcnow)
    model_used: str
    prompt_tokens: int = Field(default=0, ge=0)
    completion_tokens: int = Field(default=0, ge=0)
    time_to_first_token_ms: float = Field(default=0.0, ge=0.0)
    total_stream_duration_ms: float = Field(default=0.0, ge=0.0)
    estimated_cost_usd: float = Field(default=0.0, ge=0.0)
    was_cancelled: bool = Field(default=False)

```

**Deviation from Template Rule 3:** Default temperature in reference code is `0.7` (assumed conversational creativity for a chatbot) vs. `<= 0.3` required by generic template. This is a deliberate product decision to document in the README rather than a non-compliance to fix—a conversational chatbot does not share the strict determinism requirements of a structured extraction pipeline.

---

## 4. Vector Store & RAG

**Not applicable to Project 4.** The template marks this optional; the streaming chatbot uses no vector retrieval—memory is managed via the `messages` array retained on the client side and sent back in full on each turn. Section intentionally kept blank to preserve template structure (traceability: "considered, discarded").

---

## 5. Resilience & Cache

**Adapted.** The template's caching strategy (`SHA256(input + prompt + model)` → cached payload) assumes a single replayable response. A token-by-token SSE stream cannot be replayed identically:

* **Retry / backoff (Tenacity, 4 attempts max, jitter):** Applies **only** to establishing the upstream connection (before the first `yield`). Once the first token is emitted to the client, silent retries are impossible without duplicating text already displayed—mid-stream failures emit an `event: error` (Rule 2) instead of a transparent retry.
* **Caching:** Irrelevant for a conversational chatbot with variable multi-turn context (the `SHA256(full messages)` key would almost never match twice). Caching remains useful strictly for **mock/offline mode** (deterministic demonstration responses)—to be documented as a restricted use case, not a production mechanism.

---

## 6. Tech Stack & Infrastructure Constraints

| Component | Tool / Standard | Requirement |
| --- | --- | --- |
| **Backend Language** | Python | `>= 3.11` (`mypy --strict`) |
| **Backend Framework** | FastAPI | `>= 0.110`, Starlette ASGI |
| **ASGI Server** | Uvicorn | `uvloop` + `httptools` |
| **LLM Client** | `AsyncOpenAI` / `httpx.AsyncClient` | Zero sync client inside `async def` — immediate rejection |
| **Contracts** | Pydantic V2 | Models `frozen=True`, `extra="forbid"` |
| **Resilience** | Tenacity | Exponential backoff + jitter, connection setup only (see Section 5) |
| **Quality** | Ruff + Mypy strict | Zero warning tolerance |
| **Dependency Mgmt** | Poetry | Deterministic lockfile |
| **Frontend** | React 18 + TypeScript + Vite | Strict SSE payload typing |
| **Styling** | Tailwind CSS | Utility-first |
| **Container** | Docker | Multi-stage, non-root user (`UID 10001`) |

---

## 7. Frontend Contract *(added section — absent from template)*

The template only covers the backend; however, 15% of Project 4's grade relies on client-side error/cancellation UX, requiring an explicit contract between layers:

```typescript
export interface Message {
  role: 'user' | 'assistant' | 'system';
  content: string;
}

export interface StreamPayload {
  content?: string;
  error?: string;
}

export type StreamConnectionState = 'idle' | 'connecting' | 'streaming' | 'error' | 'cancelled';

```

**Associated Rules:**

* `StreamConnectionState` must be an explicit state machine (no scattered `isLoading`/`hasError` booleans)—prevents invalid states during cancellation.
* SSE parsing (`data: {...}\n\n`) must handle truncated frames across `ReadableStream` chunks (buffering, never `JSON.parse` an unconfirmed complete line).
* One `AbortController` per active request; never share controllers across sequential submissions.

---

## 8. Real-Time Observability *(added section — expands Template Rule 4)*

The template mandates `prompt_tokens`, `completion_tokens`, `execution_time_seconds`, `estimated_cost_usd`, `is_cached`. This is insufficient for streaming, as the primary product metric (TTFT < 100 ms, 20% of evaluation) does not exist in request-response mode. Additional required metrics, supplied by `StreamFinOpsReport` (Section 3):

| Metric | Measured From | Project Target Threshold |
| --- | --- | --- |
| `time_to_first_token_ms` | Request receipt → first `event: message` | < 100 ms |
| `total_stream_duration_ms` | First → last `event: message` | Informational, no hard limit |
| `was_cancelled` | Client-side `AbortController` detected on server | Must reliably report `true`, not inferred via timeout |

---

## 9. Lifecycle & Cancellation *(added section — expands Template Rule 2)*

Absent from template as it does not model long-lived connections cancellable by clients. Three behaviors must be guaranteed:

1. **Clean Cancellation:** `client.disconnect` (Starlette `request.is_disconnected()`) must be checked periodically inside the `async for chunk in stream` loop to release upstream sockets immediately upon drop—otherwise upstream generation keeps consuming paid tokens after the user clicks "Stop".
2. **`event: end` Idempotency:** Emitted exactly once in a `finally` block, regardless of whether the stream finishes normally, errors out, or disconnects.
3. **No Resource Leaks:** The async generator must wrap the upstream loop in `try/finally` (not just the initial call)—an `except`-only block (as in current reference code) fails to guarantee cleanup if an exception hits after multiple chunks have already been emitted.