# Phased mitigation rollout — CVE-2021-22681 for a small municipal utility

**Scope + honesty note.** This is a *mitigation plan*, not a product and not a conformance claim. It
is deliberately right-sized for a **small water / wastewater utility** — a handful of staff, a tight
budget, usually a contracted O&M operator plus an integrator — **not** the full enterprise
FactoryTalk / Active-Directory / DMZ stack Rockwell's own hardening guidance assumes. Because
CVE-2021-22681 **cannot be patched** (Rockwell, PN1550: an architectural hardcoded-key flaw), the
work is **ongoing, not one-time** — which is exactly why it's a durable managed need rather than a
single engagement. Unfamiliar term below? `GLOSSARY.md`.

The single most important fact for sequencing: **the fix that fully closes the flaw (CIP Security /
per-device certs) depends on the device supporting it.** Legacy Logix often does not. So the plan is
built so that a utility gets **most of the risk reduction from Phases 0–2 regardless of hardware**,
and reaches the real cryptographic fix in Phase 3 *if* the equipment allows — with segmentation +
monitoring as the durable compensating control if it doesn't.

---

## Phase 0 — Stop the bleeding (hours to days · low/no cost)
The highest-impact, cheapest actions. Do these before anything else.
- **Get the PLC off direct internet exposure.** This is CISA's #1 recommendation and it defeats the
  remote-attacker path entirely for most victims. Remove port-forwards/NAT to the controller; require
  **VPN** for any remote engineering access.
- **Self-check exposure** (e.g. Shodan against the utility's public IP ranges) to confirm nothing
  Logix-family is internet-reachable on native ports (44818/EtherNet-IP, 2222, etc.).
- **Inventory:** every Logix-family device, firmware version, and whether it is (or was) reachable.
  You cannot protect what you have not listed. `PHASE0_INVENTORY_WORKSHEET.md` is a fillable version
  of this line — an operator and an integrator can finish it in an afternoon.
- **Change any default credentials; disable unused services/ports** on the controllers and the
  engineering workstation.

## Phase 1 — Segment (days to weeks · one switch + one firewall)
- Put OT on its **own VLAN/segment behind a firewall** — a small-utility version of the Purdue model,
  not a full multi-zone DMZ.
- **Allowlist only the required flows** (the specific CIP traffic between the HMI / engineering
  station and the PLC); **deny everything else**, especially anything crossing from the business/IT
  side or the internet. Concretely, using the ports Phase 0's exposure check already names:

  | Source | Destination | Port / Protocol | Action |
  |---|---|---|---|
  | Engineering workstation / HMI | PLC | TCP 44818 (EtherNet/IP explicit messaging) | Allow |
  | Engineering workstation / HMI | PLC | UDP 2222 (CIP I/O implicit messaging) | Allow, only if actually used |
  | Anything else, including business/IT VLAN | PLC | any | **Deny** |
  | Internet | PLC | any | **Deny** — this is Phase 0's "off direct internet exposure," enforced |

  This is illustrative, not a finished ruleset — your integrator confirms the exact ports your
  specific devices and HMI software actually use before implementing it.
- Lock down the **engineering workstation** (the machine that runs Studio 5000) — it is the highest-
  value pivot; treat it as sensitive.
- **Rockwell's own reference for this phase:** the *Converged Plantwide Ethernet (CPwE) Design and
  Implementation Guide* covers exactly this ground — segmentation, firewalls, and architectures for
  legacy equipment — in more depth than a from-scratch plan can:
  https://literature.rockwellautomation.com/idc/groups/literature/documents/td/enet-td001_-en-p.pdf
  Rockwell pointed to this document directly (PSIRT reply, 2026-08-03) when asked what to do for
  hardware that can't reach CIP Security. Treat it as the authoritative architecture reference; this
  phase is the small-utility sequencing layer on top of it, not a competing plan.

## Phase 2 — Compensating controls, because it's unpatchable (weeks · modest cost)
For the residual risk that segmentation doesn't cover — and as the **durable control if the hardware
can't do CIP Security**:
- **Passive monitoring** on a mirror/SPAN port: a lightweight IDS (Zeek/Snort) or even a small
  CIP-write alerter that flags *unexpected writes* to control tags. The attack in the wild is an
  unauthorized **write** to a controller; alerting on anomalous writes is high-signal.
- **Strict change control** on who may connect to the controller and when.
- **This control has a lifespan — know it now.** Once Phase 3 enables CIP Security, that traffic is
  encrypted and this alerter goes **blind**. It is not a permanent capability that "Monitoring"
  in Phase 4 can just assume still works the same way; see Phase 3 for what replaces it.

## Phase 3 — CIP Security / PKI: the real fix (weeks to months · hardware-permitting)
This is the layer the PoC in this repo demonstrates the *principle* of. **Only available if the Logix
hardware/firmware supports CIP Security** — verify per device. Real, tested commands for everything
below: `PHASE3_CA_QUICKSTART.md`. Questions to hand your integrator instead of running the PoC
yourself: `INTEGRATOR_CHECKLIST.md`.

- **Prerequisite, easy to miss: time synchronization.** Certificate validity windows and "the
  *current* CRL" both depend on devices agreeing what time it is. OT equipment often has drifted
  clocks and no NTP discipline — confirm this before anything below, or the PKI fails in ways that
  look like unrelated network trouble.
- **Stand up a lightweight internal CA** — issuance genuinely is a few lines; see
  `PHASE3_CA_QUICKSTART.md`. If already in the RA ecosystem, FactoryTalk Policy Manager is the
  vendor path.
- **Issue per-device certificates** and enable **mutual TLS with identity binding** — the controller
  authenticates *which* engineering station, not merely that a cert is CA-signed (Test 3 principle:
  identity in the SAN, and a negative control proving the identity check is load-bearing).
- **Honest about scope: issuance is the easy part; revocation and rotation are not.** Neither is "a
  few lines" via raw `openssl` the way issuance is — they need a real workflow or a purpose-built
  tool (`step-ca`/Smallstep, or FactoryTalk Policy Manager). This phase *requires* revocation and
  Phase 4 *requires* rotation, so budget for the full lifecycle, not just standing up the CA.
- **Configure revocation (CRL or OCSP; or short-lived certs) — and decide, in writing, what happens
  when the revocation source is unreachable.** Real fork, no universally correct answer:
  **fail-closed** stops a compromised credential but can also lock out a legitimate engineer during
  an outage that has nothing to do with security — a loss-of-control event caused by the control
  itself. **Fail-open** keeps access available but means revocation is bypassable by anyone who can
  block the CRL fetch. **Default for a continuously-run process:** favor short-lived certificates
  (hours-to-days, not months) so an unrevoked-but-compromised credential expires quickly regardless
  of CRL state; fail-open on CRL unavailability so a security control doesn't double as an
  availability control; and **alert loudly** the moment CRL checks start failing so any fail-open
  window is known and short, not silent and indefinite. Make this decision on purpose, for your own
  risk tolerance — don't let it default itself.
- **Name the trade-off this phase makes with Phase 2:** enabling CIP Security encrypts the traffic
  Phase 2's write-alerter was reading — **that alerter goes blind here**. Replace the visibility
  rather than losing it silently: monitor at the point TLS terminates (the PLC or HMI's own
  endpoint), use controller-side audit logging if the device has it, or make a deliberate, named
  decision to accept metadata-only visibility (connection timing/volume, not content) in exchange
  for the stronger authentication.
- **Honest caveat:** if the equipment predates CIP Security, this phase is not available — Phases 1–2
  are then the *permanent* compensating control, and firmware upgrade / device replacement becomes a
  budget-planning item, not an emergency.

## Phase 4 — Operate + maintain (ongoing · the durable need)
Because the flaw never gets patched away, the safe state must be *maintained*:
- **Revocation process:** who revokes a credential, and when (engineer offboarding, device
  decommission, suspected compromise). This is the operational half of Phase 3's cryptography.
- **Rotation schedule — dates on an actual calendar, well before expiry, not after.** Device certs
  at 825 days and a 10-year root (`PHASE3_CA_QUICKSTART.md`) give this plant a certificate-expiry
  failure mode it never had before Phase 3: an expired device cert means the engineering workstation
  can no longer reach the PLC, at whatever hour that happens to fall — and 825 days is almost
  exactly long enough that whoever set it up may not be the one present when it fires.
  `test5_rotation.py`'s own case 5 already proves the safe pattern: issue the replacement *before*
  retiring the old one, so identity never loses access during the transition. Alert well ahead of
  each expiry date, not on it.
- **Monitoring + an incident-response runbook** sized for the utility (a one-page playbook, not a
  SOC) — `INCIDENT_RESPONSE_QUICK_REFERENCE.md` is a starting point for the first 60 minutes; build
  your actual runbook with your primacy agency and an integrator, don't stop at this card. **If
  Phase 3 is in place, "monitoring" is not Phase 2's original write-alerter** (see Phase 3) —
  confirm what actually still has visibility before assuming the word means the same thing it did
  two phases ago.
- **Periodic re-exposure check** (repeat the Phase 0 Shodan/self-scan on a schedule).
- **Know who's already offering free help:** `RESOURCES.md` lists CISA, EPA's technical assistance
  program, WaterISAC, and AWWA — several of Phases 1–4 can be done *with* one of these rather than
  alone.

---

## What this plan is honest about
- **It is a plan, not a deployment.** Real rollout needs the specific device inventory and each
  device's CIP-Security capability; those determine whether Phase 3 is reachable.
- **CIP Security is hardware-gated.** The plan is explicitly built to deliver value without it, so a
  utility on legacy gear is not left with "nothing until you buy new PLCs."
- **Staffing reality.** Every step is intended to be achievable by a **small O&M operator + an
  integrator**, not a dedicated security team. Where a step needs specialist help (CA setup, IDS
  tuning), it is called out as an integrator task, not assumed in-house.

*l0gic — Patrick Crosby, 2026-07-31. Sequencing follows
CISA advisory AA26-097A guidance (remove PLCs from direct internet exposure first) and Rockwell PN1550
(CIP Security as the vendor-stated remediation). Verify device-specific CIP-Security support and all
vendor citations against primary sources before advising a specific utility.*