--- name: resilience description: > Resilience patterns for .NET 10 applications using Polly v8. Covers retry, circuit breaker, timeout, fallback, rate limiter, hedging, and composing resilience pipelines. Load this skill when implementing retry logic, circuit breakers, handling transient failures, or when the user mentions "Polly", "resilience", "retry", "circuit breaker", "timeout", "fallback", "rate limit", "hedging", "transient fault", "HttpClient resilience", or "resilience pipeline". --- # Resilience ## Core Principles 1. **Polly v8 resilience pipelines, not v7 policies** — Polly v8 replaced `Policy` with `ResiliencePipeline`. Never use `PolicyBuilder`, `Policy.Handle<>()`, or `ISyncPolicy`. The new API is composable, type-safe, and integrates natively with `IHttpClientFactory`. 2. **Configure via `AddResilienceHandler`, not manual wrapping** — For HTTP calls, use `Microsoft.Extensions.Http.Resilience` which adds pipelines directly to `HttpClient` via DI. No manual `ExecuteAsync` wrapping. 3. **Compose strategies, don't nest them** — A single `ResiliencePipeline` can chain retry + circuit breaker + timeout. Strategies execute outer-to-inner (first added = outermost). No need for nested try/catch or manual orchestration. 4. **Always set timeouts** — Every external call needs a timeout. Use Polly's `AddTimeout()` as the innermost strategy so it applies per-attempt, and optionally an outer timeout for total elapsed time. 5. **Instrument everything** — Polly v8 emits `Metering` events and supports `TelemetryOptions` for OpenTelemetry. Use them to monitor retry rates, circuit breaker state, and timeout frequency. ## Patterns ### HTTP Client Resilience (Recommended Default) ```csharp // Program.cs — Standard resilience handler covers 90% of use cases builder.Services.AddHttpClient(client => { client.BaseAddress = new Uri("https://api.payments.example.com"); }) .AddStandardResilienceHandler(); // Retry + circuit breaker + timeout out of the box // That's it. The standard handler configures: // - Retry: 3 attempts, exponential backoff, jitter // - Circuit breaker: 10% failure ratio over 30s sampling, 30s break // - Attempt timeout: 10s per attempt // - Total request timeout: 30s ``` **Why**: `AddStandardResilienceHandler()` from `Microsoft.Extensions.Http.Resilience` applies production-ready defaults. Override only when you need different thresholds. ### Custom HTTP Resilience Configuration ```csharp builder.Services.AddHttpClient(client => { client.BaseAddress = new Uri("https://api.catalog.example.com"); }) .AddResilienceHandler("catalog", builder => { // Total timeout — outermost, caps total elapsed time builder.AddTimeout(TimeSpan.FromSeconds(15)); // Retry — exponential backoff with jitter builder.AddRetry(new HttpRetryStrategyOptions { MaxRetryAttempts = 3, BackoffType = DelayBackoffType.Exponential, UseJitter = true, Delay = TimeSpan.FromMilliseconds(500), ShouldHandle = static args => ValueTask.FromResult( args.Outcome.Result?.StatusCode is HttpStatusCode.RequestTimeout or HttpStatusCode.TooManyRequests or HttpStatusCode.ServiceUnavailable || args.Outcome.Exception is HttpRequestException) }); // Circuit breaker — prevent cascading failures builder.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions { FailureRatio = 0.5, SamplingDuration = TimeSpan.FromSeconds(10), MinimumThroughput = 10, BreakDuration = TimeSpan.FromSeconds(30) }); // Per-attempt timeout — innermost builder.AddTimeout(TimeSpan.FromSeconds(5)); }); ``` **Why**: Named resilience handlers let you tune per-service. The order matters: total timeout > retry > circuit breaker > attempt timeout. ### Non-HTTP Resilience Pipeline ```csharp // For database calls, message queues, or any non-HTTP operation builder.Services.AddResiliencePipeline("database", builder => { builder .AddRetry(new RetryStrategyOptions { MaxRetryAttempts = 3, BackoffType = DelayBackoffType.Exponential, Delay = TimeSpan.FromMilliseconds(200), ShouldHandle = new PredicateBuilder() .Handle() .Handle(ex => ex.Message.Contains("deadlock", StringComparison.OrdinalIgnoreCase)) }) .AddTimeout(TimeSpan.FromSeconds(10)); }); // Inject and use public sealed class OrderRepository( AppDbContext db, [FromKeyedServices("database")] ResiliencePipeline pipeline) { public async Task GetByIdAsync(Guid id, CancellationToken ct) { return await pipeline.ExecuteAsync( async token => await db.Orders.FindAsync([id], token), ct); } } ``` **Why**: `AddResiliencePipeline` registers a named pipeline in DI. Inject with `[FromKeyedServices]` for clean, testable code. ### Typed Resilience Pipeline ```csharp // When the operation returns a specific type, use ResiliencePipeline builder.Services.AddResiliencePipeline("external-api", builder => { builder .AddFallback(new FallbackStrategyOptions { FallbackAction = static args => { var response = new HttpResponseMessage(HttpStatusCode.OK) { Content = new StringContent("{\"status\":\"degraded\",\"data\":[]}") }; return Outcome.FromResultAsValueTask(response); }, ShouldHandle = static args => ValueTask.FromResult( args.Outcome.Exception is not null || args.Outcome.Result?.IsSuccessStatusCode == false) }) .AddRetry(new RetryStrategyOptions { MaxRetryAttempts = 2, Delay = TimeSpan.FromMilliseconds(500) }) .AddTimeout(TimeSpan.FromSeconds(5)); }); ``` **Why**: Typed pipelines let you add fallback strategies that return a default value when all retries are exhausted — critical for graceful degradation. ### Hedging (Parallel Requests) ```csharp builder.Services.AddHttpClient() .AddResilienceHandler("search-hedging", builder => { builder.AddHedging(new HttpHedgingStrategyOptions { MaxHedgedAttempts = 2, Delay = TimeSpan.FromMilliseconds(500) // Send parallel request after 500ms }); builder.AddTimeout(TimeSpan.FromSeconds(3)); }); ``` **Why**: Hedging sends a parallel request if the first hasn't responded within the delay. Use for latency-sensitive reads where you can tolerate duplicate work. ### Telemetry Integration ```csharp builder.Services.AddResiliencePipeline("monitored", (builder, context) => { // Polly v8 emits metrics via System.Diagnostics.Metrics automatically. // ConfigureTelemetry wires structured logging for strategy events. builder .ConfigureTelemetry(new TelemetryOptions { LoggerFactory = context.ServiceProvider.GetRequiredService() }) .AddRetry(new RetryStrategyOptions { MaxRetryAttempts = 3 }) .AddCircuitBreaker(new CircuitBreakerStrategyOptions()) .AddTimeout(TimeSpan.FromSeconds(10)); }); // In Program.cs — wire up OpenTelemetry to capture Polly metrics builder.Services.AddOpenTelemetry() .WithMetrics(metrics => metrics.AddMeter("Polly")); ``` ### Rate Limiting (.NET Built-in) .NET provides built-in rate limiting middleware via `AddRateLimiter()` — no external packages needed. Algorithms: `AddFixedWindowLimiter`, `AddSlidingWindowLimiter`, `AddTokenBucketLimiter`, `AddConcurrencyLimiter`. ```csharp builder.Services.AddRateLimiter(options => { options.AddFixedWindowLimiter("fixed", opt => { opt.PermitLimit = 100; opt.Window = TimeSpan.FromSeconds(60); opt.QueueLimit = 0; }); // Always return ProblemDetails with Retry-After on 429 options.OnRejected = async (context, ct) => { context.HttpContext.Response.StatusCode = StatusCodes.Status429TooManyRequests; if (context.Lease.TryGetMetadata(MetadataName.RetryAfter, out var retryAfter)) context.HttpContext.Response.Headers.RetryAfter = ((int)retryAfter.TotalSeconds).ToString(); await context.HttpContext.Response.WriteAsJsonAsync( new ProblemDetails { Title = "Too many requests", Status = 429 }, ct); }; }); app.UseRateLimiter(); app.MapGet("/api/orders", ListOrders).RequireRateLimiting("fixed"); ``` ## Anti-patterns ### BAD: Using Polly v7 API ```csharp // BAD — v7 policy syntax, do not use var retryPolicy = Policy .Handle() .WaitAndRetryAsync(3, attempt => TimeSpan.FromSeconds(Math.Pow(2, attempt))); var response = await retryPolicy.ExecuteAsync(() => httpClient.GetAsync("/api/data")); ``` ### GOOD: Polly v8 Resilience Pipeline ```csharp // GOOD — v8 pipeline via DI builder.Services.AddHttpClient() .AddStandardResilienceHandler(); ``` --- ### BAD: Wrapping Every Call Manually ```csharp // BAD — manual resilience per call site public async Task GetOrderAsync(Guid id) { try { return await _pipeline.ExecuteAsync(async ct => await _httpClient.GetFromJsonAsync($"/orders/{id}", ct)); } catch (TimeoutRejectedException) { return Order.Empty; } catch (BrokenCircuitException) { return Order.Empty; } } ``` ### GOOD: Pipeline Handles Everything via HttpClient DI ```csharp // GOOD — resilience is configured at the HttpClient level public async Task GetOrderAsync(Guid id, CancellationToken ct) { var response = await _httpClient.GetAsync($"/orders/{id}", ct); if (!response.IsSuccessStatusCode) return null; return await response.Content.ReadFromJsonAsync(ct); } ``` --- ### BAD: Retry on Non-Idempotent Operations ```csharp // BAD — retrying a POST that creates a resource risks duplicates builder.AddRetry(new RetryStrategyOptions { MaxRetryAttempts = 5 // This will create 5 orders on transient failures! }); ``` ### GOOD: Retry Only Idempotent Operations or Use Idempotency Keys ```csharp // GOOD — use idempotency key header for non-idempotent operations builder.AddRetry(new HttpRetryStrategyOptions { MaxRetryAttempts = 3, ShouldHandle = static args => ValueTask.FromResult( args.Outcome.Result?.StatusCode is HttpStatusCode.RequestTimeout or HttpStatusCode.ServiceUnavailable) }); // Pair with idempotency key in the request httpClient.DefaultRequestHeaders.Add("Idempotency-Key", Guid.NewGuid().ToString()); ``` --- ### BAD: Circuit Breaker Without Monitoring ```csharp // BAD — circuit breaker with no visibility into state changes builder.AddCircuitBreaker(new CircuitBreakerStrategyOptions()); // How do you know when it trips? You don't. ``` ### GOOD: Circuit Breaker with Telemetry ```csharp // GOOD — Polly v8 metrics captured via OpenTelemetry builder.Services.AddOpenTelemetry() .WithMetrics(metrics => metrics.AddMeter("Polly")); // Dashboard alerts on: polly.circuit_breaker.state = Open ``` ## Decision Guide | Scenario | Strategy | Configuration | |----------|----------|---------------| | HTTP calls to external APIs | `AddStandardResilienceHandler()` | Use defaults, override only specific thresholds | | HTTP with custom thresholds | `AddResilienceHandler("name", ...)` | Named handler with per-service tuning | | Database / EF Core calls | `AddResiliencePipeline("db", ...)` | Retry on deadlock/timeout, no circuit breaker | | Message queue publishing | `AddResiliencePipeline("mq", ...)` | Retry with exponential backoff, timeout | | Latency-sensitive reads | `AddHedging(...)` | Parallel request after delay threshold | | Graceful degradation | `AddFallback(...)` | Return cached/default value on total failure | | Per-attempt time limit | `AddTimeout(...)` innermost | 2-10s depending on operation | | Total operation time limit | `AddTimeout(...)` outermost | Sum of all retries + buffer | | Non-idempotent writes | Retry with idempotency key | Or no retry — fail fast | | Read-heavy microservice | Standard handler + hedging | Low latency with redundancy | | API rate limiting | `AddRateLimiter()` + `RequireRateLimiting()` | Fixed, sliding, or token bucket per endpoint |