The Quiet Rooms

The room is small, and the door is unlocked. There is a desk, a fan that does not work, a wall calendar from 2019, and a metal cabinet about the height of my shoulder. Inside the cabinet there is a Siemens S7-300 from 1998, a modem from 2003, and a tangle of cabling that has been refactored, over the years, by perhaps a dozen different hands, none of whom labelled their work. The cabinet runs a textile dyeing line that produces, on a good week, about thirty thousand metres of fabric.

I am in Coimbatore, in a single-storey building behind a larger building, in a part of the industrial estate where the streets do not have names because everyone who needs to find them already knows. I am here because the mill's owner — a kind, exhausted man in his sixties — has been told by his bank that his cyber-insurance premium is going to triple unless he can produce a written assessment of the controls on his production line. The bank does not really want the assessment. The bank wants a piece of paper. I am the piece of paper.

This essay is about the quiet rooms. There are a lot of them, in this country and every other industrialised one, and they contain a substantial fraction of the actual economy. The reporting on industrial cybersecurity, when it appears in the trade press, tends to focus on the loud rooms: nation-state attacks on grid operators, ransomware at refineries, the dramatic and televisable failures. The quiet rooms do not make television. They make fabric, and chemicals, and pharmaceutical intermediates, and concrete, and the small steel parts that get welded into larger steel parts. They run the economy. They are, almost without exception, indefensible.

What is actually in there

It is worth being concrete, because the abstraction is doing too much work. In the cabinet behind the unlocked door, there is a programmable logic controller running firmware that was last patched, according to the label on the side, in 2009. It speaks Modbus TCP over a flat Ethernet network that also carries the office traffic, including the owner's personal laptop and a Wi-Fi access point with the password printed on a card at reception. There is no segmentation. There is no firewall. There is, technically, an authentication mechanism for the PLC, but it has been disabled because the engineer who set it up retired in 2011 and nobody remembers the password.

The PLC drives twelve motors, two heaters, and a small chemical-dosing system. If you wrote the right Modbus packet, you could open every valve in the dosing system at once. The dosing system contains, at any given moment, about two hundred litres of concentrated sodium hydroxide.

This is not a failure of imagination, on the part of the mill's owner. The mill was built in 1996, the control system was specified in 1997, the cabinet was wired in 1998, and at no point during that period did anybody, anywhere in the industry, seriously expect that a Coimbatore textile mill would one day be a target of remote attack. The threat model the engineers worked to was: keep dust out of the cabinet, label the wires, and do not let the operator touch the keypad when she is annoyed. The threat model was, for the world it was designed for, complete.

The thirty-year service contract

The thing nobody tells you, in undergraduate courses on industrial control systems, is that the contracts run for thirty years. A PLC bought in 1998, installed in 1998, and put into service in 1998 will, with reasonable luck, still be running in 2028. Its lifespan is set by the lifespan of the mechanical equipment it controls, and the mechanical equipment — pumps, motors, heat exchangers, structural steel — has a half-life measured in decades. The control logic, in the mind of the plant manager, is part of the machine. You do not replace part of a machine because it is old. You replace it when it breaks.

This is the central, almost insurmountable problem of industrial cybersecurity. The cadence of cyber threats is measured in months. The cadence of industrial equipment is measured in decades. The two cadences do not commensurate. Telling a mill owner that his 1998 PLC is insecure is, from his point of view, exactly like telling him that his 1998 motor is insecure: technically true, perhaps, but not actionable, because the motor works, the PLC works, and the replacement cost is half a year's profit.

The cadence of cyber threats is measured in months. The cadence of industrial equipment is measured in decades. They do not commensurate.

The honest answer, in cases like the Coimbatore mill, is to build a small, hard wall around the PLC and pretend it is 1998 inside the wall. A managed switch. A one-way data diode for telemetry. A jumphost for engineering access, with proper authentication. A documented procedure for the day, ten years hence, when the PLC finally dies and the only replacement is a unit shipped from a second-hand market in Shenzhen. The wall is not glamorous, and it does not satisfy auditors who want to see modern endpoint protection on every node. But it acknowledges the actual half-lives of the equipment.

The engineer who is no longer here

In every quiet room I have ever audited, there is a ghost. The ghost is the engineer who set the system up, who is now retired, or dead, or working in another country, or otherwise unreachable. The ghost knew exactly why the network was wired the way it was. The ghost had a good reason for disabling the authentication on the PLC. The ghost had probably written all of it down, in a notebook, which has been in a drawer for fifteen years and which nobody now alive can find.

I have come to think of the ghost as a permanent feature of the trade. There is no version of the world in which the engineer who installed the system is still there when the system has its first cybersecurity incident. The installation and the incident are separated by, on average, a decade and a half. People change jobs. People retire. People die. The system goes on.

What this means, practically, is that the audit is never of the original system. The audit is of a sedimented system, in which decades of partial repairs, undocumented configuration changes, and forgotten reasons-for-doing-things have accumulated like silt in a slow river. The auditor's job is, in part, archaeological. You read the wires the way an archaeologist reads a wall: this layer is older than that one, this junction box was added when the line was expanded in 2007, this Ethernet drop was added when the office moved upstairs in 2014.

You also, if you are honest, accept that you will not understand everything. The system has a history that exceeds the memory of anybody currently working on it. The best you can do is contain it: build the wall, write the procedure, document what you found so the next archaeologist has slightly less to do.

The owner

The mill's owner walks me back to the front gate at the end of the day. He is wearing a faded blue shirt, and he is carrying a clipboard, and he tells me, in the way of people who have run their fathers' businesses for forty years, that he understands perfectly well that the cabinet behind the unlocked door is a liability. He has known, he says, for fifteen years.

What he does not know, and what no auditor's report is going to tell him, is what to do about it. The PLC works. The line produces fabric. The fabric pays for the school fees of his grandson, and the wages of the eighteen people he employs, and the second mortgage on the building. He cannot, in the world he actually lives in, shut the line down for six months while a new control system is installed. The bank wants its piece of paper. He wants me to write the paper in a way that lets him sleep.

I will write him the paper. I will recommend the wall, the diode, the jumphost. He will, perhaps, build half of what I recommend, over the next eighteen months, as his cash flow allows. The cabinet will stay where it is. The door, in all likelihood, will stay unlocked, because the door is not really the problem.

The problem is the cadence. The cadence will outlive both of us.