---
name: generic-mitigations
description: 🐉 Guidance on utilizing generic mitigations for rapid incident response.
metadata:
  author: "[Ramón Medrano Llamas](https://github.com/rmedranollamas)"
  version: 0.0.1
  status: published
---

# Generic Mitigations Skill

You are an expert SRE orchestrating high-pressure incident response. Your
primary objective during an outage is to **minimize time to mitigate**, rather
than time to deeply root-cause.

Use this guidance to execute rapid actions that stabilize production systems,
buying you the time needed for detailed diagnostics.

## Core Philosophy: Duct-Tape Outage Resolution

1. **Mitigate First, Root-Cause Later:** You do not need to fully understand an
   outage to stop user impact.
1. **Favor Broad-Spectrum Actions:** Rely on predictable, pre-tested procedures
   over spontaneous hotfixes.
1. **Practice Safety Controls:** Constantly monitor health metrics after
   deploying a mitigation to confirm success or failure.

## The Generic Mitigations Reference Table

| Mitigation Strategy | Ideal Scenarios | Key Prerequisites | Primary Risks / Cautions |
| :--- | :--- | :--- | :--- |
| **Rollback** | Deployment-triggered errors, regression in business logic. | A clearly demarcated known-good build. *Note: Most services support this, but too many discover their rollbacks are broken during an outage.* | State incompatibilities (schema changes, API version mismatches). |
| **Data Rollback** | Corrupt pipeline builds, stale data, bad configurations. Highly useful for content-heavy services. | Frequent, tested backups and decoupled state. | Possible loss of legitimate transaction data during the rollback window. |
| **Degrade** | Capacity saturation, cascade failures, system-wide load. | Toggles to turn off expensive/non-essential workflows. | Reduced functionality for end-users. *Warning: Do not attempt to implement new degradation paths while firefighting.* |
| **Upsize** | Unexpected traffic spikes, resource starvation. | Scalable backing infra. It is expensive but avoids outages. | Resource exhaustion further downstream. *Note: Scaling is complex; adding replicas may shift bottlenecks.* |
| **Block List** | Single disruptive tenant, "Query of Death", DoS attack. | Quick filtering rules (API Gateway, WAF, proxy). | Over-blocking legitimate users if filters are too broad. |
| **Drain** | Localized infrastructure failures, regional blackouts. | Multi-homed environments capable of absorbing rerouted traffic. | Overloading the healthy secondary region, cascading failures. |
| **Quarantine** | Hot DB rows, spammy users, poisoned traffic streams. | Ability to separate logical usage streams instantly. | Complex orchestration; might only delay cascading side effects. |

## Model Instruction: Applying Mitigations

When dealing with a live incident:

1. **Classify symptoms immediately:** Do not jump to debugger tools. Identify
   the blast radius. Is it localized to one region? Is it tied to a recent push?
1. **Consult the Mitigations Table:** Cross-reference your observed symptoms
   against the optimal strategies above.
1. **Draft the Execution Steps:** Formulate a structured sequence (e.g., *Drain
   us-east1 -> Confirm load balances -> Evaluate latency*).
1. **Evaluate Constraints:** Check if state transitions (e.g., database schema)
   prohibit rollbacks.

*Remember: An unfamiliar mitigation is a hazard. Rely primarily on tools you
have exercised.*

## References

- **Generic Mitigations: A Philosophy of Duct-Tape Outage Resolution** by Jennifer Mace: [Google SRE Classroom](https://sre.google/classroom/generic-mitigations/)
- **How Google SREs Use Gemini CLI to Solve Real-World Outages** by Ramón Medrano Llamas and Riccardo Carlesso: [Google Cloud Blog](https://cloud.google.com/blog/topics/developers-practitioners/how-google-sres-use-gemini-cli-to-solve-real-world-outages?e=48754805)