Firewall change plan with rollback for OT sites

An OT firewall change plan lists, for every rule change, the finding it closes, the flows it must keep, the exact commands or console steps, the order to apply them, the pre-checks, how to verify it worked, and a rollback that puts the rule back as found. Write the rollback before the change, apply it in a planned window that operations knows about, add the named allows before you narrow or disable a broad rule, log what you drop so a missed dependency shows up at once, and confirm the result with the next traffic log. NIST SP 800-82r3 section 6.2.4.2 asks for changes to be documented, tested and approved before they reach the OT environment.

A rule change in OT is a production change. A rule that looks unused can carry the one session a line needs at shutdown, and a broad rule can hide a dependency nobody wrote down. NIST SP 800-82r3 section 2.4 notes that OT outages are planned days or weeks in advance, and that change management needs OT experts such as control engineers working with security and IT staff. The plan is how those people agree on a change before anyone touches the firewall.

Change plan checklist

Each change in the plan carries:

1. The finding it closes, with the sessions and the rule behind it.
2. The flows it keeps, from the traffic log: source, destination, application, and who needs them.
3. The rule as found, copied from the configuration export, so the rollback has something exact to restore.
4. The change in the firewall’s own commands and as console steps, naming zones, hosts and applications.
5. The risk: what could stop if the traffic log missed a flow, and who would notice first.
6. Pre-checks: a fresh backup of the configuration, the rule still matching the export, its hit count and last hit read again, operations told.
7. Verification: what to watch during the window, and what the next traffic log should show.
8. The rollback: the commands that restore the rule as found, and the signs that call for it.
9. The approval: the change ticket or management-of-change record, and who signed it. In the SP 800-53 OT overlay, CM-3 (configuration change control) says change control procedures should align with the organization’s management of change practices.

Order the changes safely

The order inside one change matters as much as the change itself.

1. Check the unused allows below the broad rule. They match nothing today only because the broad rule matches first. Narrow the broad rule and the next one down becomes the live path, so disable them before or with the change.
2. Add the named allows, above the broad rule, for the flows production needs.
3. Add a logged deny for the rest of that path.
4. Disable the broad rule rather than deleting it. Disabling keeps the rollback to one command.

Across changes, work one boundary at a time, and send a change that repeats across sites to a pilot site first. Inside OT, between cells and levels, start with rules that log rather than block, and tighten only once the log shows the conduit carries nothing unexpected. Never remove cyclic I/O or a safety path outside a planned window with a rollback ready. The east-west guide covers why.

Write the rollback first

A rollback is the change undone in reverse order: re-enable the original rule, then remove what the change added. Written before the window, it can be checked against the configuration export, so it restores the rule as found rather than as someone remembers it.

The illustrative PAN⁠-⁠OS change on the SegAudit home page shows the pattern. The change keeps one path, RDP from the IDMZ jump hosts to a Level 3 host, logs and drops the rest of the path into a Level 1 cell, and disables the old rule:

set rulebase security rules l35-ops-eng from dmz-l35 to ops-l3
set rulebase security rules l35-ops-eng source jump-hosts destination ops-a-01
set rulebase security rules l35-ops-eng application ms-rdp service application-default
set rulebase security rules l35-ops-eng action allow log-end yes
set rulebase security rules l35-cell-a-drop from dmz-l35 to cell-a-l1 source any destination any
set rulebase security rules l35-cell-a-drop application any service any action deny log-end yes
move rulebase security rules l35-ops-eng before dmz-to-cell-any
move rulebase security rules l35-cell-a-drop after l35-ops-eng
set rulebase security rules dmz-to-cell-any disabled yes

Its rollback is three lines:

set rulebase security rules dmz-to-cell-any disabled no
delete rulebase security rules l35-cell-a-drop
delete rulebase security rules l35-ops-eng

Agree before the window what calls for the rollback and who decides: an HMI that loses its controllers, an alarm operations don’t expect, a session the process needs showing up in the deny log, or the window running out. Delete the disabled rule in a later window, once the retest shows nothing needed it.

During and after the window

• Watch the logged deny. A session there from a host production needs is a missed dependency. Add a named rule for it, or roll back.
• Check with operations. HMIs, historians and alarms should look normal before the window closes.
• Retest with the next traffic log. Each fixed path should be closed, and each kept flow should still pass. A path that still carries traffic means the change was not applied or something else allows it.
• Update the record. The zone map, the rule review sheet and the change ticket all change with the rules. The retest is also the evidence an auditor asks for; see proving OT segmentation.

How SegAudit builds the change plan

SegAudit’s change package turns each finding into a change for the site’s own change process: least-privilege allows for the flows production uses, logged drops for everything else, and for each change a risk assessment, firewall commands, console steps, pre-checks, verification and rollback. Rules repeated across sites are handled once, with each site’s own addressing, and global changes go to a pilot site first. Each site’s plan carries safety checks and says whether it has a blocking failure before anything goes to production. The commands are written for Panorama and PAN⁠-⁠OS, FortiGate, Cisco ASA, Cisco FTD through FMC, and Check Point.

In the Northline sample (a made-up three-site company that ships with SegAudit), CHG-06 replaces the any-any rule allow-any-temp at three sites. Its rollback reads: “Delete the 3 new rules and re-enable allow-any-temp at each site (one commit and push, minutes). The objects can stay.” Its timing reads: “Planned maintenance window with operations on site. Pilot one site first, keep the rollback staged, and watch through a full production cycle before the next site.” The rollback commands also note that a later change, CHG-08, relies on this change’s drop, so that one is rolled back first.

After the window, load the next traffic log and each change comes back Verified, Regressed, Not applied or No traffic.

The commands are a suggestion and can disable production rules. Stage them in a lab or a non-production device group first, and follow your change advisory process. SegAudit never connects to the firewall and pushes nothing.

To see what SegAudit catches in one of your own exports, run the free Quick check. The sample deliverable shows a full change package.

Common questions

Should an OT firewall change go through management of change?

Yes. A rule change can stop production as surely as a logic change. NIST SP 800-82r3 section 6.2.4.2 asks for changes to be documented, tested and approved before deployment, and the SP 800-53 OT overlay ties configuration change control (CM-3) to the organization’s management of change practices.

Should we delete or disable the old rule?

Disable it first. A disabled rule makes the rollback one command. Delete it in a later window, once a retest over the site’s normal cycle shows nothing needed it.

What if there is no traffic log?

Then the plan cannot show which flows to keep. Turn logging on for the rules in scope and collect a window first. NIST SP 800-82r3 section 6.2.1.3 suggests temporarily allowing and recording traffic between segments to learn which communication to authorize.

How long should we watch after the change?

At least one full shift, and through the next batch run or maintenance job the rule might serve. Keep the deny logging on, so anything missed shows up in the log rather than as a silent failure.