# Ralph Circuit Breaker — Model Rate Limit Fallback

> Classic circuit breaker pattern (Hystrix / Polly / Resilience4j) applied to Copilot model selection.
> When the preferred model hits rate limits, Ralph automatically degrades to lower-cost models, then self-heals.

## Problem

When running multiple Ralph instances across repos, Copilot model rate limits cause cascading failures.
All Ralphs fail simultaneously when the preferred model (e.g., `claude-sonnet-5`) hits quota.

Under usage-based billing, all models incur cost — there are no "free" or unlimited models.
Lightweight-category models have lower per-token cost and are therefore preferred for fallback,
but they are still billed:

| Model | Category | Cost profile |
|-------|----------|--------------|
| `claude-sonnet-5` | versatile | Moderate — standard-tier default |
| `claude-opus-4.8` | powerful | Higher — reserve for premium work |
| `gpt-5.4` | powerful | Higher — standard-tier specialist |
| `gpt-5.4-mini` | lightweight | Lower-cost — preferred fallback |
| `gpt-5-mini` | lightweight | Lowest-cost — final fallback |

## Circuit Breaker States

```
┌─────────┐   rate limit error    ┌────────┐
│ CLOSED  │ ───────────────────►  │  OPEN  │
│ (normal)│                       │(fallback)│
└────┬────┘   ◄──────────────── └────┬────┘
     │        2 consecutive          │
     │        successes              │ cooldown expires
     │                               ▼
     │                          ┌──────────┐
     └───── success ◄────────  │HALF-OPEN │
             (close)            │ (testing) │
                                └──────────┘
```

### CLOSED (normal operation)
- Use preferred model from config
- Every successful response confirms circuit stays closed
- On rate limit error → transition to OPEN

### OPEN (rate limited — fallback active)
- Fall back through the lightweight-category model chain:
  1. `gpt-5.4-mini`
  2. `gpt-5-mini`
- Start cooldown timer (default: 10 minutes)
- When cooldown expires → transition to HALF-OPEN

### HALF-OPEN (testing recovery)
- Try preferred model again
- If 2 consecutive successes → transition to CLOSED
- If rate limit error → back to OPEN, reset cooldown

## State File: `.squad/ralph-circuit-breaker.json`

```json
{
  "state": "closed",
  "preferredModel": "claude-sonnet-5",
  "fallbackChain": ["gpt-5.4-mini", "gpt-5-mini"],
  "currentFallbackIndex": 0,
  "cooldownMinutes": 10,
  "openedAt": null,
  "halfOpenSuccesses": 0,
  "consecutiveFailures": 0,
  "metrics": {
    "totalFallbacks": 0,
    "totalRecoveries": 0,
    "lastFallbackAt": null,
    "lastRecoveryAt": null
  }
}
```

## PowerShell Functions

Paste these into your `ralph-watch.ps1` or source them from a shared module.

### `Get-CircuitBreakerState`

```powershell
function Get-CircuitBreakerState {
    param([string]$StateFile = ".squad/ralph-circuit-breaker.json")

    if (-not (Test-Path $StateFile)) {
        $default = @{
            state              = "closed"
            preferredModel     = "claude-sonnet-5"
            fallbackChain      = @("gpt-5.4-mini", "gpt-5-mini")
            currentFallbackIndex = 0
            cooldownMinutes    = 10
            openedAt           = $null
            halfOpenSuccesses  = 0
            consecutiveFailures = 0
            metrics            = @{
                totalFallbacks  = 0
                totalRecoveries = 0
                lastFallbackAt  = $null
                lastRecoveryAt  = $null
            }
        }
        $default | ConvertTo-Json -Depth 3 | Set-Content $StateFile
        return $default
    }

    return (Get-Content $StateFile -Raw | ConvertFrom-Json)
}
```

### `Save-CircuitBreakerState`

```powershell
function Save-CircuitBreakerState {
    param(
        [object]$State,
        [string]$StateFile = ".squad/ralph-circuit-breaker.json"
    )

    $State | ConvertTo-Json -Depth 3 | Set-Content $StateFile
}
```

### `Get-CurrentModel`

Returns the model Ralph should use right now, based on circuit state.

```powershell
function Get-CurrentModel {
    param([string]$StateFile = ".squad/ralph-circuit-breaker.json")

    $cb = Get-CircuitBreakerState -StateFile $StateFile

    switch ($cb.state) {
        "closed" {
            return $cb.preferredModel
        }
        "open" {
            # Check if cooldown has expired
            if ($cb.openedAt) {
                $opened = [DateTime]::Parse($cb.openedAt)
                $elapsed = (Get-Date) - $opened
                if ($elapsed.TotalMinutes -ge $cb.cooldownMinutes) {
                    # Transition to half-open
                    $cb.state = "half-open"
                    $cb.halfOpenSuccesses = 0
                    Save-CircuitBreakerState -State $cb -StateFile $StateFile
                    Write-Host "  [circuit-breaker] Cooldown expired. Testing preferred model..." -ForegroundColor Yellow
                    return $cb.preferredModel
                }
            }
            # Still in cooldown — use fallback
            $idx = [Math]::Min($cb.currentFallbackIndex, $cb.fallbackChain.Count - 1)
            return $cb.fallbackChain[$idx]
        }
        "half-open" {
            return $cb.preferredModel
        }
        default {
            return $cb.preferredModel
        }
    }
}
```

### `Update-CircuitBreakerOnSuccess`

Call after every successful model response.

```powershell
function Update-CircuitBreakerOnSuccess {
    param([string]$StateFile = ".squad/ralph-circuit-breaker.json")

    $cb = Get-CircuitBreakerState -StateFile $StateFile
    $cb.consecutiveFailures = 0

    if ($cb.state -eq "half-open") {
        $cb.halfOpenSuccesses++
        if ($cb.halfOpenSuccesses -ge 2) {
            # Recovery! Close the circuit
            $cb.state = "closed"
            $cb.openedAt = $null
            $cb.halfOpenSuccesses = 0
            $cb.currentFallbackIndex = 0
            $cb.metrics.totalRecoveries++
            $cb.metrics.lastRecoveryAt = (Get-Date).ToString("o")
            Save-CircuitBreakerState -State $cb -StateFile $StateFile
            Write-Host "  [circuit-breaker] RECOVERED — back to preferred model ($($cb.preferredModel))" -ForegroundColor Green
            return
        }
        Save-CircuitBreakerState -State $cb -StateFile $StateFile
        Write-Host "  [circuit-breaker] Half-open success $($cb.halfOpenSuccesses)/2" -ForegroundColor Yellow
        return
    }

    # closed state — nothing to do
}
```

### `Update-CircuitBreakerOnRateLimit`

Call when a model response indicates rate limiting (HTTP 429 or error message containing "rate limit").

```powershell
function Update-CircuitBreakerOnRateLimit {
    param([string]$StateFile = ".squad/ralph-circuit-breaker.json")

    $cb = Get-CircuitBreakerState -StateFile $StateFile
    $cb.consecutiveFailures++

    if ($cb.state -eq "closed" -or $cb.state -eq "half-open") {
        # Open the circuit
        $cb.state = "open"
        $cb.openedAt = (Get-Date).ToString("o")
        $cb.halfOpenSuccesses = 0
        $cb.currentFallbackIndex = 0
        $cb.metrics.totalFallbacks++
        $cb.metrics.lastFallbackAt = (Get-Date).ToString("o")
        Save-CircuitBreakerState -State $cb -StateFile $StateFile

        $fallbackModel = $cb.fallbackChain[0]
        Write-Host "  [circuit-breaker] RATE LIMITED — falling back to $fallbackModel (cooldown: $($cb.cooldownMinutes)m)" -ForegroundColor Red
        return
    }

    if ($cb.state -eq "open") {
        # Already open — try next fallback in chain if current one also fails
        if ($cb.currentFallbackIndex -lt ($cb.fallbackChain.Count - 1)) {
            $cb.currentFallbackIndex++
            $nextModel = $cb.fallbackChain[$cb.currentFallbackIndex]
            Write-Host "  [circuit-breaker] Fallback also limited — trying $nextModel" -ForegroundColor Red
        }
        # Reset cooldown timer
        $cb.openedAt = (Get-Date).ToString("o")
        Save-CircuitBreakerState -State $cb -StateFile $StateFile
    }
}
```

## Integration with ralph-watch.ps1

In your Ralph polling loop, wrap the model selection:

```powershell
# At the top of your polling loop
$model = Get-CurrentModel

# When invoking copilot CLI
$result = copilot-cli --model $model ...

# After the call
if ($result -match "rate.?limit" -or $LASTEXITCODE -eq 429) {
    Update-CircuitBreakerOnRateLimit
} else {
    Update-CircuitBreakerOnSuccess
}
```

### Full integration example

```powershell
# Source the circuit breaker functions
. .squad-templates/ralph-circuit-breaker-functions.ps1

while ($true) {
    $model = Get-CurrentModel
    Write-Host "Polling with model: $model"

    try {
        # Your existing Ralph logic here, but pass $model
        $response = Invoke-RalphCycle -Model $model

        # Success path
        Update-CircuitBreakerOnSuccess
    }
    catch {
        if ($_.Exception.Message -match "rate.?limit|429|quota|Too Many Requests") {
            Update-CircuitBreakerOnRateLimit
            # Retry immediately with fallback model
            continue
        }
        # Other errors — handle normally
        throw
    }

    Start-Sleep -Seconds $pollInterval
}
```

## Configuration

Override defaults by editing `.squad/ralph-circuit-breaker.json`:

| Field | Default | Description |
|-------|---------|-------------|
| `preferredModel` | `claude-sonnet-5` | Model to use when circuit is closed |
| `fallbackChain` | `["gpt-5.4-mini", "gpt-5-mini"]` | Ordered fallback models (lightweight-category — lowest cost, still billed) |
| `cooldownMinutes` | `10` | How long to wait before testing recovery |

## Metrics

The state file tracks operational metrics:

- **totalFallbacks** — How many times the circuit opened
- **totalRecoveries** — How many times it recovered to preferred model
- **lastFallbackAt** — ISO timestamp of last rate limit event
- **lastRecoveryAt** — ISO timestamp of last successful recovery

Query metrics with:
```powershell
$cb = Get-Content .squad/ralph-circuit-breaker.json | ConvertFrom-Json
Write-Host "Fallbacks: $($cb.metrics.totalFallbacks) | Recoveries: $($cb.metrics.totalRecoveries)"
```