The Long Half-Life of a Bad Default

The router on the desk is small, beige, and ten years old. Its plastic case has yellowed in the way of all small beige things from the mid-2010s. On the bottom, in twelve-point type, is a sticker that reads: Username: admin / Password: admin. Below that, a serial number, an FCC ID, and a single line of compliance microprint nobody has ever read. It is unremarkable in every respect, which is exactly the point.

I was given the router by a friend of a friend, a paediatrician in Pune who runs a small clinic out of a rented first-floor office. The clinic had been hit, mildly, by something that asked for two thousand dollars in bitcoin and then, when ignored, went quietly away. The clinic's IT, such as it was, consisted of the router, a printer, two laptops, and a Wi-Fi camera mounted above the reception desk. I came to look at the router.

What I found, in the way these things go, was less a vulnerability than a sediment. Every firmware update the router had ever received, going back to 2014, sat in the device's flash memory like layers of paint in an old house. Each layer had been authored by a different team, at a different company, working from different assumptions about what the device was for. None of them had ever removed the telnet daemon that had been there from the start.

Sediment

There is a particular kind of decision, in the early life of a product, that nobody ever returns to. The engineer who first enables telnet on a development board does so because telnet is convenient, and because the device, at that point, is one of three on a desk, plugged into a switch that goes nowhere. The decision is correct in its context. It survives the move to the lab. It survives the first internal demo. It survives the first customer trial, where the support team finds that being able to telnet in is a marvellous diagnostic shortcut. By the time anyone thinks to question it, the device is shipping by the thousand, the support team has built tooling around it, and the original engineer has left the company.

This is how bad defaults persist. Not through villainy, but through the ordinary, slightly tired physics of large organisations. Every removal is a cost; every status quo is, by definition, free. The telnet daemon stays.

The chip never lies. But it does remember every decision you ever made about it, and it tells anyone who asks.

I have been writing firmware, in one form or another, for fourteen years. The single most useful thing I can tell a young embedded engineer is that nothing they ship is temporary. The firmware they push to a development board on a Tuesday afternoon, intending to rewrite it on Wednesday, will be in a hospital basement in 2034. The thing they typed in a comment as a joke will be read, eventually, by a lawyer. The default password they used because the QA team kept locking themselves out is the default password forever.

The shape of a long failure

There are, broadly, three kinds of firmware failure I encounter in the field. The first is acute: a bug, a regression, a memory corruption that takes the device down in front of the customer. These are the failures that get fixed, because they generate phone calls, and phone calls generate engineering tickets, and tickets generate, eventually, an apologetic note to release. Acute failures are loud and short-lived. They are the easy ones.

The second kind is what I think of as a slow failure. The device works. It does the thing it was sold to do. But it works in a way that is, over time, corrosive: it consumes more battery than it should, or it generates twice as many writes to flash as the bill of materials anticipated, or it leaks a small amount of memory on every reboot. The slow failure shows up two years in, when devices in the field start dying at twice the predicted rate. By then the team that wrote the firmware is on the next product.

The third kind, the kind I want to talk about, is the latent failure. The latent failure is a decision, made early, that does nothing wrong for years and then becomes catastrophic the moment the environment changes. The telnet daemon is a latent failure. The factory password is a latent failure. The hard-coded private key embedded in the bootloader because somebody, in 2015, needed to sign an OTA update and the production CA wasn't ready yet — that is a latent failure of the most painful sort, because the device is doing exactly what it was designed to do, and what it was designed to do is now wrong.

---

You can spot latent failures in code reviews, sometimes, if you know what to look for. The shape is always the same: a comment that says TODO: remove before production, with no date attached. A configuration flag whose default is true because the developer was tired. A debug endpoint that returns the full contents of memory if you pass the right magic string in the URL, which the developer assured everyone was just for the bring-up team.

I once saw a piece of bootloader code, in a fairly well-known smart lock, that contained the comment // XXX: this is fine for now, fix when we move to production. The comment had been there for six years and through three acquisitions. The code it described was, by any reasonable engineering standard, not fine.

What the router told me

The Pune router, when I finally got a serial console attached, revealed itself slowly. The boot log was long and chatty, in the way of devices that were never expected to have anyone reading them. It announced its kernel version (3.10.49, from 2014), its U-Boot version, the SPI flash chip it had detected, the size of its RAM, and — almost in passing, halfway down the third screen of text — that it had started a telnet daemon on port 23, bound to the LAN interface, with the username and password set to the values printed on the sticker.

This is the part of the story where the audience usually expects me to explain how I exploited the device. The explanation is short and unsatisfying. I typed telnet 192.168.1.1, I entered admin twice, and I had a root shell. The thing that took me three hours, in the end, was not getting in. It was figuring out, from the layered sediment of firmware updates, which team had been the last to think about the device.

The answer, as best I could reconstruct it, was that nobody had. The original manufacturer had been acquired in 2018 by a larger consumer-electronics company, which had been acquired in 2021 by a private-equity vehicle. The firmware repository, when I finally tracked down a former engineer through LinkedIn, had been migrated twice and then quietly retired. There was no team. There was nobody to write to. The router, in any meaningful sense, was an orphan.

Living with sediment

I do not think there is a clean solution to this, and I am suspicious of anyone who says there is. The economic incentives of consumer electronics push, very hard, towards shipping a thing and then forgetting about it. The regulatory environment, in most jurisdictions, does not require manufacturers to support devices for the life of their physical use. The buyer of a five-dollar smart plug is not, in any reasonable sense, paying for a decade of patches.

What I can say, from inside the firmware-writing trade, is that the most useful thing we can do is to be honest about half-lives. When you make a decision, write down how long you expect it to live. If the answer is two weeks, set yourself a calendar reminder. If the answer is forever, design accordingly: no default credentials, no debug endpoints in shipping firmware, no telnet on anything that touches a network. If the answer is "until somebody fixes it," assume nobody will, and design for that world instead.

The Pune router is now disconnected from the network. The clinic has a new one, a recent model from a different vendor, configured by me, with telnet off and a password that did not come printed on a sticker. The old router sits on my desk. I look at it sometimes, when I am writing firmware for some new device, and I try to imagine the engineer who will, in ten years, be holding my work in their hands, wondering what on earth I was thinking.

The chip never lies. But it does remember every decision you ever made about it, and it tells anyone who asks.