# Resilience Best Practices

## Overview

This reference provides design guidance, naming conventions, operational patterns, and anti-patterns for AWS resilience services. Use as a companion to the procedural skills when making architectural decisions or establishing team practices.

## Parameters

**tier** (optional): Service tier for policy recommendations (0, 1, 2, or 3)
**service_count** (optional): Number of services being onboarded (affects organizational guidance)

## Policy Design

### Tiered Policy Model

**Constraints:**

- You MUST recommend a tiered policy model (classify services by business criticality) rather than one policy per service
- You MUST set the availability SLO higher as criticality rises. The API accepts only a fixed set of SLO values and rejects out-of-set ones — **confirm the valid values from the API/docs** (e.g. `aws resiliencehubv2 create-policy help`) rather than relying on a hardcoded list (illustratively, values such as `99.9`/`99.95`/`99.99`).
- You MUST tighten RTO/RPO as criticality rises (single-digit minutes for critical, hours for low)
- You MUST match the DR approach to criticality (more aggressive for more critical services), using a value from the API's DR-approach enum — **verify the valid set via the API/docs** (e.g. `aws resiliencehubv2 create-policy help`); illustratively `ACTIVE_ACTIVE` … `BACKUP_AND_RESTORE`.
- Example (illustrative, not a fixed table): critical payments/auth → `99.99` + low-minutes RTO + `ACTIVE_ACTIVE`; standard internal tools → `99.9` + tens-of-minutes RTO + `WARM_STANDBY`; low dev/test → `99.9` + multi-hour RTO + `BACKUP_AND_RESTORE`. Treat the RTO figures as rough illustration, not commitments — very aggressive multi-AZ RTOs (≈1 minute) are realistic only for clean, hard failures; gray/partial failures are slower to detect and recover, so set targets you can actually meet under realistic failure modes.

### Policy Anti-Patterns

**Constraints:**

- You MUST warn against these anti-patterns:
  - Setting the maximum SLO (99.99) with Backup & Restore DR (contradictory)
  - Same policy for all services (over-engineering low-tier services)
  - Multi-region RTO < multi-AZ RTO (region failover is always slower)
  - No policy at all (assessments have no target to measure against)

## System & Service Organization

### Naming Conventions

**Constraints:**

- You SHOULD recommend consistent naming:
  - Systems: `{domain}-platform` (e.g., payment-platform, identity-platform)
  - Services: `{domain}-{component}` (e.g., payment-api, payment-processor)
  - Journeys: `{verb}-{noun}` (e.g., process-payment, authenticate-user)
  - Policies: `tier-{N}-{qualifier}` (e.g., tier-1-standard, tier-0-critical)

### Tagging Strategy

**Constraints:**

- You MUST recommend resilience-specific tags that serve double duty (NGRH discovery + FIS targeting):
  - `resilience:tier` — Policy tier (0-3)
  - `resilience:system` — Parent system name
  - `resilience:service` — Service name
  - `resilience:owner` — Owning team
  - `resilience:fis-enabled` — Whether FIS experiments are authorized ("true"/"false")

## Input Source Strategy

**Constraints:**

- You MUST recommend input sources in priority order:
  1. CloudFormation stacks — Most reliable, tracks drift
  2. Terraform state — Good for multi-cloud shops
  3. EKS clusters — For containerized workloads
  4. Resource tags — Catch-all for resources outside IaC
  5. Design files — For planned (not yet deployed) architectures

## FIS Experiment Hygiene

### Progressive Complexity

**Constraints:**

- You MUST recommend progressive experiment complexity:
  - Week 1: Single-component, single-fault (terminate one instance)
  - Month 1: Multi-component, single-fault (AZ failure affecting multiple services)
  - Quarter 1: Multi-fault, cascading (AZ failure + increased load)
  - Ongoing: Surprise experiments (unannounced to on-call team)
- You MUST enforce that every experiment has stop conditions — never run without them
- You MUST recommend enabling FIS experiment logging (`--log-configuration` to CloudWatch Logs or S3) for every experiment to capture action-execution detail for audit and post-experiment analysis; the log destination SHOULD be KMS-encrypted since experiment logs can reveal topology and failure modes
- You SHOULD recommend naming: `{service}-{scenario}-{scope}` (e.g., checkout-api-az-failure-us-east-1a)

### Stop Condition Design

**Constraints:**

- You MUST recommend tight stop conditions for production: error rate > 1% for 1 minute
- You SHOULD recommend looser conditions for GameDays: error rate > 10% for 5 minutes (allow team to respond)
- You MUST NOT allow experiments without stop conditions in any environment

## ARC Operational Patterns

### Zonal Autoshift Readiness

**Constraints:**

- You MUST verify these prerequisites before enabling autoshift:
  - Application handles 33% capacity reduction (one of three AZs gone)
  - No AZ-pinned resources (EBS volumes, placement groups)
  - Health checks detect AZ-scoped failures
  - Practice run outcome alarm configured
- You MUST NOT enable autoshift if the application cannot survive losing one AZ

### Routing Control Discipline

**Constraints:**

- You MUST always require safety rules (never allow all-off state)
- You MUST recommend documenting the failover runbook (who can flip controls, under what conditions)
- You SHOULD recommend testing failover monthly (use FIS to trigger, ARC to recover)
- You SHOULD recommend enabling and monitoring CloudTrail across the whole lifecycle for a complete audit trail — routing control state changes (ARC), `fis:StartExperiment`/`fis:StopExperiment` (FIS), and `resiliencehubv2` assessment/finding operations (NGRH)

## Operational Cadence

**Constraints:**

- You MUST recommend this minimum cadence:
  - Continuous: ARC zonal autoshift practice runs
  - Weekly: Review NGRH findings dashboard
  - Monthly: Run FIS experiments (single-service)
  - Quarterly: Cross-service GameDay
  - After incidents: Re-assess affected services, add new experiments
  - After deployments: Verify routing controls and autoshift still healthy

## Anti-Patterns to Avoid

**Constraints:**

- You MUST warn against these anti-patterns when encountered:
  - "We'll add resilience later" — Build it in from day one
  - Testing only in non-prod — Production has different failure modes
  - Manual failover only — Automate with ARC routing controls
  - Assessing once and forgetting — Architecture drifts, re-assess regularly
  - Ignoring achievability=NOT_ACHIEVABLE — Fix architecture before testing
  - FIS without stop conditions — Unbounded blast radius
  - ARC without safety rules — Risk of turning off all traffic
  - Marking findings resolved without FIS validation — Paper compliance; validate the fix with a fault-injection experiment BEFORE marking the finding resolved, never after