Skip to main content
OT/ICS Cybersecurity

Anatomy of a Step an Operator Can Follow at 2 a.m.: Rewriting the OT Incident Response Procedure

"Isolate the affected system as appropriate" hides five decisions and hands them to the night shift. The five parts of an executable step, trigger, actor, action, check and way out, with three before-and-after rewrites and a fifteen-minute console walk-through to prove them.

Papercraft scene in a dark control room at night: a crumpled strip of paper stands beside a neat stack of five coloured paper blocks, each cut with an icon for alarm, person, gear, checklist and exit, under a desk lamp

"Isolate the affected system and notify the relevant people as appropriate." Read that line the way a night-shift operator reads it: alone at the console at 2 a.m., with an alarm she has never seen before. Which system? Isolate it how, and from where? Who said she could? Who are the relevant people, and what does "appropriate" mean tonight?

Count the questions. That one line hides five decisions and hands every one of them to the person with the least information and the least time. It reads perfectly well at a desk in daylight, which is where it was written and reviewed.

The cost of a hidden decision has been measured. The US Department of Energy's Human Performance Improvement Handbook (DOE-HDBK-1028-2009, Volume 1, pages 2-24 to 2-26) puts the chance of error in familiar, rule-based work at about 1 in 1,000. In unfamiliar work, where people reason from first principles under stress, it puts the odds at roughly 1 in 2 to 1 in 10. The figures come from nuclear-industry studies and are orders of magnitude. A cyber incident on a control system is about as unfamiliar as plant work gets. Every decision you settle on paper moves the operator from the second column toward the first.

This is for whoever has been handed the response pages of an OT incident plan, or an IT plan someone wants to "make work for the plant". Everything below needs a pen, your current plan and, at the end, one operator for fifteen minutes.

Five parts, and no decisions left over

Sandia National Laboratories publishes a free Technical Procedures Writer's Guide (SAND2021-0153, January 2021) for high-consequence facility work. Its rules for action steps carry straight across to an incident plan. Boiled down, a step an operator can follow cold has five parts.

Two rules from Degani and Wiener's NASA study of flight-deck checklists (CR-177549, 1990) sharpen the template. A response should state the value the reader expects, so "FW-OT-2 rule set = ISLAND" beats "firewall: done". And the most critical items go first, before the first likely interruption, even when logical order says otherwise. A control room in an incident is interrupted every minute.

Here is the opening line run through the template. The site is a fictional bottling plant we will call Larch Hill, and every tag, register row and time limit below is invented for the example.

Before: "Isolate the affected system and notify the relevant people as appropriate."

After:

  1. Call the incident lead (contacts register, row C-01) and ask: "Do you approve isolating FW-OT-2?"
  2. IF the incident lead approves, THEN go to step 3. WHEN 15 minutes pass with no answer, THEN the shift supervisor decides under pre-authorization PA-1 in the plan's authority register.
  3. Performed by: controls technician. Set firewall FW-OT-2 to rule set ISLAND.
  4. Confirm that alarm LINK-DOWN is active on HMI-3, screen Plant Links. IF LINK-DOWN is not active within 2 minutes, THEN call the integrator support line (contacts register, row V-02) and stay on this step.
  5. Write the time of isolation in the shift log. Call the IT security on-call (row C-04) and say: "Larch Hill has isolated FW-OT-2 at [time]."

Sequence complete. Report to the incident lead when they take over.

Look at what each change removed. "Affected system" became a named firewall. "Isolate" became one action with a named rule set. "As appropriate" became an approval with a timed fallback. "Relevant people" became two register rows and the exact sentence to say. And the operator now has a way to tell that it worked, which the original never offered. Longer on the page, faster at the console.

Two more rewrites, for traps the first one hides

An action buried in a note. Before: "NOTE: The engineering workstation should not be restarted, and memory should be captured first." That note holds two actions and no performer. After:

CAUTION: Restarting or logging off EWS-2 erases the contents of its memory, which the investigation may need.

  1. Leave EWS-2 powered on and logged in.
  2. Performed by: OT forensic lead. Capture the memory of EWS-2: GO TO evidence procedure OT-SOP-11, section 3, then RETURN TO this procedure, step 3.

A condition that reads two ways. Before: "If the HMI is frozen and the PLC is not responding or the safety system has tripped, shut down." Read it twice. Does a safety trip on its own mean shutdown, or only alongside a frozen screen? The sentence supports both readings. Sandia's guide says never to mix AND with OR in one condition, and to switch to a table once there are three or more. After:

What you seeWhat you do
The safety system has trippedGO TO emergency shutdown procedure ESD-01
HMI-3 is frozen AND the PLC-FILL fault light is on at panel LP-4GO TO emergency shutdown procedure ESD-01
HMI-3 is frozen AND the PLC-FILL fault light is offRun the filler from panel LP-4 under the normal manual procedure, then continue at step 6

GO TO means the reader does not come back. GO TO with RETURN TO means they do. Sandia calls every jump an error opportunity, so use one only when the content cannot sensibly sit in the step, and always give the procedure ID and section. "Follow site procedures" is a jump with no destination.

A quick test: could a stranger name the device your worst step acts on, from the step alone? If not, you have found where to start.

"They have run this plant for twenty years"

My operators know this plant better than anyone. They do not need a paragraph telling them which firewall to touch. Spell it out like that and they will feel talked down to.

They know the plant in its normal state. An incident step is the opposite of routine: rare, complex and high in consequence. Sandia's guide asks for more detail for exactly those tasks, less only for frequent work done by qualified staff, and it gives a rule for when you are unsure: write to the lowest level. At 2 a.m. the reader may be the newest qualified operator, a holiday contractor, or the veteran with a phone ringing in each ear.

Write for the least experienced qualified person who could be alone on shift. The veteran loses nothing. He reads "FW-OT-2", recognises it and acts. Precision is not condescension: "FW-OT-2" is shorter than "the appropriate firewall", and it only means one thing.

"I do not know the plant's words"

I write security policy. I don't know the tag names, the screen names, or who is allowed to approve isolation at night. How am I supposed to write steps like that?

You are the editor here, not the source of the plant's facts. Write the draft with every unknown in square brackets, [firewall tag], [screen name], [approver], and then take it to the console. Sit a night-shift operator down with the draft and ask them to say out loud what they would press, read and call at each step. Listen for four things:

  • Can they find every named device, screen and phone without leaving the console?
  • Does every trigger describe something they can actually see from there?
  • Does any step need a key, a password or a tool they do not hold at night?
  • Is any step one they would read a different way from the one you meant?

Sandia recommends the same for any procedure: a reviewer from the intended audience who writes down how they read each step, because the gap between writer and user is a major source of error. This checks the document; it is not an exercise. Can you name the operator you would ask? If not, that is your first gap, and it is not a writing gap.

"It will be out of date by spring"

We change something every month. A new integrator, a firewall swap, a reshuffled phone tree. Any step that names a tag will be wrong by spring, so why bother being that specific?

A specific step goes stale where you can see it. A vague step is wrong all the time, invisibly. Three habits keep it honest. Keep each changing fact in one master place, such as the contacts register by row, and point at it instead of retyping a phone number. Put the revision and date on every page. And tie the procedure to the plant's change process, so a firewall swap triggers a review of every step that names it. Searching the plan for "FW-OT-2" takes seconds. Searching for "the appropriate firewall" finds nothing.

Decisions belong in daylight

Every decision left inside a step still gets made. The only question is when, and by whom. Leave it in the step and it gets made at 2 a.m. by the operator with the least information, in the mode where errors run one in a handful. Write the trigger, the actor, the action, the check and the way out, and the same decision gets made weeks earlier, by people with time, drawings and the authority to make it. The one-line isolation step looked efficient. It was the slowest sentence in the plan.

Sources: US DOE, Human Performance Improvement Handbook, DOE-HDBK-1028-2009, Vol. 1 | Sandia National Laboratories, Technical Procedures Writer's Guide, SAND2021-0153 | Degani and Wiener, Human Factors of Flight-Deck Checklists: The Normal Checklist, NASA CR-177549

ICS/OT Incident Response Plan: Write One That Works

Most OT incident response plans fail before anyone writes a word: no sponsor, no charter, no agreement on who decides. This workshop course takes you from a blank page to a plan your site approves and your operators can follow.

Each of the eight sections writes one chapter of your plan: the authority and scope, the inputs, severity in the language of the plant, command and decision rights, response steps, recovery and the readiness library.

You get the plan template, more than 40 templates and worksheets, 21 worked examples and a capstone with a full model answer.

50 videos, 7.7 hours. US$147 one-time through HKSM Courses. 30-day money-back guarantee when 50% or less consumed.

Explore the Course


ICS/OT Incident Response Plan course preview

Write the OT incident plan your site will actually use

Go from a blank page to an approved, usable plan for your industrial site, one chapter at a time. ICS/OT Incident Response Plan: Write One That Works includes 50 videos, the plan template, worksheets and a full model answer, delivered through HKSM Courses for US$147 one-time. 30-day money-back guarantee when 50% or less of the course has been consumed.

Explore the Course