Skip to main content
OT/ICS Cybersecurity

When the HMI Stops Answering: The Safe-State Sheet Your OT Incident Plan Is Missing

In July 2026 attackers changed PLC passwords to lock out water plant operators, and some utilities ran by hand. Your operators know every safe state; your plan probably does not. A nine-column collection sheet, one filled row, and who to ask, so the plan says what the plant already knows.

Painterly night scene of an operator in a hard hat turning a handwheel at a lit local control panel of gauges and switches, a dark monitor and treatment tanks behind him

Four words in a federal alert should stop every plan writer for a minute. On 30 July 2026, CISA told water utilities that attackers had changed PLC passwords "to lock out operators".

The same alert says the activity led to boil water notices and "sustained manual operations". In Minnesota, more than 30 community water systems were disrupted, and some utilities switched to running their equipment by hand. A password change took the screens away. It could not take away what the operators knew: where each process had to end up, and how to get it there with the controls in front of them.

That knowledge is the most valuable input an OT incident response plan has, and in most plans it is nowhere. It lives in the heads of people on days, in an emergency shutdown procedure written for a pump failure, and in a cause-and-effect chart in the engineering office. This post is for the person writing the plan, who already has operators who know the plant better than any document does, and who needs the plan to say what they know before a night when the HMI stops answering. The tool is one sheet, and none of the answers on it will be yours.

The sheet, and the people who fill it

The standard behind this is short. IEC 62443-2-1 (published in the US as ANSI/ISA-62443-2-1-2024), requirement AVAIL 1.3, asks for procedures to bring the control system to "a predetermined state if normal operation cannot be maintained as a result of a detected security compromise". The word that matters for a writer is predetermined. The state exists before the incident, or it is agreed before the plan is approved. Nobody works it out at 3 a.m.

Here is one row filled in, for a chemical dosing skid at a water plant invented for this article. The procedure numbers, hours and staffing are illustrative.

ColumnIllustrative entry
UnitChlorine dosing skid, train A
Safe state"Train A dosing at the last manual setpoint, train B isolated, residual checked by hand every 30 minutes"
Who may order itShift supervisor; nights: lead operator on call
Without the HMI?Yes: hand-off-auto switch on the local panel in the chemical building, handheld residual meter in the lab cupboard
Manual endurance72 hours, two additional operators each shift
Forces a shutdownResidual cannot be read for more than one hour; chlorine leak alarm
Existing procedureOP-114, Loss of SCADA, chemical feed
Practised in last year?No: finding F-07, passed to whoever runs exercises
SignedProcess engineer, name and date

Two columns on that row do most of the work. The first is the route without the screen. A reply like "we just switch to hand" skips the hard part, so get the switch and the meter by name, then walk out to the panel beside the operator and see them. If the only route to the safe state runs through the HMI, write that down as a gap. The second is manual endurance, which is a headcount disguised as a duration. "Indefinitely, with two more people each shift" sounds reassuring until the call-out list has a single spare name on it, and then the honest figure is one shift. Count names on the roster before you commit to hours.

Here is the test the whole sheet is built to pass: could tonight's shift name the safe state for each unit, and reach it, without the HMI? If you can answer yes for every unit and point to where it is written, you can skip ahead to the action plan. If any answer is "probably", the next three sections are for you.

"Our operators already know all this."

My crew has kept this place running for two decades. They know exactly what to do if the screens go. A spreadsheet of what they already know is paperwork, and nobody will read it.

They probably do know, and the sheet is not written for them. It is written for the relief operator on a holiday weekend, for the incident commander deciding whether to order the safe state, and for whoever has to check that manual endurance outlasts the time the control system will take to come back. Those people cannot ask a twenty-year operator who is asleep.

The interviews also find something the operators cannot see from inside their own heads: disagreement. Put one question to the engineer and the operator in the same room, and their answers often differ. Every difference is a finding, and it is far cheaper to find it at a table in daylight than at the panel during an incident.

"I'm the security person. I can't define a safe state."

I came from IT security. I'm not qualified to say where a chlorine skid should end up, and I'm not going to pretend I am.

Good, because nobody is asking you to. The people who design and run the process own that answer; you gather it. That is not a lesser job. Your skill is the next question, the one a specialist stops asking because the answer feels obvious to them: "How, exactly?", "Who on nights?", "With how many people?", "Which procedure number?". Write what the engineer says in the engineer's words, and get the signature in the last column. The signature is what makes the row an engineering decision rather than your guess.

Do you know who would sign for your most critical unit today? If you cannot name them, that is the first interview to book, before any of the others.

"Operations owns those procedures. Security can't touch them."

We already have emergency shutdown and loss-of-control procedures. Operations wrote them and operations owns them. They are not going to let the security team rewrite them for a cyber plan.

Agreed, and leave them alone. Reference each one by its number and keep its steps where they live. Copied steps drift out of date with the first revision, and a second, drifting copy of a safety procedure is a hazard of its own. The incident response plan adds only what the procedure lacks: the cyber condition that calls for it, and the person allowed to call for it. That is a small, specific ask, and operations can usually agree to it in one meeting.

What the operators in July already had

When the screens were taken away, the utilities that kept water moving did it with people who knew where each process had to go and could get it there by hand. That knowledge was not created during the incident. It existed before, in people and in procedures, and the operators reached for it.

An incident response plan cannot hold that knowledge for them. It can do something narrower and more useful: say exactly where each unit's safe state lives on paper, who may order it, how it is reached without a screen, how long the people can hold it, and which of those answers nobody has tested yet.

Sources: CISA, alert on activity targeting PLCs in the water and wastewater sector, 30 July 2026 | BleepingComputer, CISA warns of cyberattacks disrupting US water utilities, 31 July 2026 | ISA, the ISA/IEC 62443 series (ANSI/ISA-62443-2-1-2024, requirement AVAIL 1.3)

ICS/OT Incident Response Plan: Write One That Works

Most OT incident response plans fail before anyone writes a word: no sponsor, no charter, no agreement on who decides. This workshop course takes you from a blank page to a plan your site approves and your operators can follow.

Each of the eight sections writes one chapter of your plan: the authority and scope, the inputs, severity in the language of the plant, command and decision rights, response steps, recovery and the readiness library.

You get the plan template, more than 40 templates and worksheets, 21 worked examples and a capstone with a full model answer.

50 videos, 7.7 hours. US$147 one-time through HKSM Courses. 30-day money-back guarantee when 50% or less consumed.

Explore the Course


ICS/OT Incident Response Plan course preview

Write the OT incident plan your site will actually use

Go from a blank page to an approved, usable plan for your industrial site, one chapter at a time. ICS/OT Incident Response Plan: Write One That Works includes 50 videos, the plan template, worksheets and a full model answer, delivered through HKSM Courses for US$147 one-time. 30-day money-back guarantee when 50% or less of the course has been consumed.

Explore the Course