Systems Fail in Transition

An airplane is stable in the air and stable on the ground. The dangerous part is the transition between those states.

I return to that image because it separates two activities that are often treated as if they were the same: running software and operating a system.

Software can be running. A deployment can be complete. The architecture can be redundant and the dashboards can show green. The system can still be failing.

Operating begins with that uncomfortable gap. It is concerned not only with whether the software is doing what it says it is doing, but with what happens around it: how the system entered its present state, what it depends on, how it will leave, and what will happen when something has to change.

The point where the declarative meets the imperative is the dangerous part.

The interval between intent and reality

Declarative systems let us describe the state we want. That is useful. It can also make the distance between a declaration and its operational consequence easy to overlook.

The imperative is where machines, networks, storage, time, and people have to carry the declaration into reality. It is the interval in which an upgrade is underway but not complete, a new configuration is present but not proven, a replacement has begun but the old dependency still exists, or a recovery path is being used for the first time under pressure.

When software and deployments are working together, they can resemble a beautifully choreographed dance. The choreography is convincing partly because its rough spots are hidden. When something needs to change, especially at the edges, the movement becomes visible.

A transition exposes assumptions that a stable state allows us to leave unexamined. We discover which guarantees belonged to the design and which belonged only to the conditions under which we had observed it. We discover whether “replaceable” meant replaceable in theory or replaceable under operating pressure, while some other system is waiting.

This is not an argument that stable states are safe or that every change is a failure waiting to happen. It is an observation about where operating reality tends to become legible. A system can conceal its conditions while it remains in a familiar state. Movement makes those conditions harder to ignore.

What must endure

Networking and storage remain the edges I distrust most, not because they are exotic, but because they are expected to disappear into the background. They should just work. When they do, the rest of the system is free to pretend they are not there.

They are also where the language of ephemerality reaches its limit.

No matter how much we try to be idempotent, no matter how carefully we treat machines like cattle rather than pets, some things must endure. Operating knowledge must endure. Irreplaceable data must endure. Networking, like the air we breathe, must endure.

The machine may be replaceable. The dependency is not necessarily replaceable on the same terms. The difference matters during a transition, when the old thing is no longer intact and the new thing has not yet acquired all of the conditions that made the old thing work.

“Cattle, not pets” can be a useful correction to the habit of nursing individual machines forever. It is less useful when it becomes a way to avoid asking what the herd depends on, what cannot be recreated, or how long replacement actually takes. Ephemerality is a property of a component. It is not a promise that the system around it has no memory.

The enduring thing may not be the host or the process. It may be the data, the recovery path, the operating knowledge, the responsibility, or the dependency no inventory has recorded. Replacement works only when we know which of those must cross the transition with us.

Green while failing

I learned this early through BIND and DNS.

Early versions of BIND could start and appear to be running with catastrophic errors inside the configuration. Entire sections of critical records serving email and web pages could be hidden, while the software still demonstrated that the changes had been accepted and reported itself updated.

From one point of view, the change worked. From the point of view that mattered, the system was failing.

This was my first corporate job, at a large financial services and disaster recovery company. We ran bastion hosts and split DNS, internal and external. Zone files were kept in sync with rcp over ssh, and a Perl script massaged the addresses between the two views. BIND 4 checked configuration syntax weakly enough that a missing trailing period or a broken SOA could blackhole an entire domain. Two accidents usually saved you. Forgetting to increment the serial number meant the change never propagated. A sufficiently low TTL meant the mistake expired before it spread far.

One day neither saved me.

It was a routine DNS update. It took email down for the entire company for up to four hours. Not one division. Every line of business, including several that had nothing to do with technology.

The failure did not present as an outage. It presented as silence. Replies people were waiting on never arrived, because the senders’ mail servers had queued them and then began returning messages saying our domain was not configured. The broken path was not one we operated or watched. It was somebody else’s mail server, forming an opinion about us and telling our correspondents.

I told my team and my boss immediately. Owning a mistake early and disclosing it is the only part of this story I would repeat unchanged. It escalated quickly to the CEO, who worked in the same building.

It is not until the CEO is storming up and down the hallways shouting, “Who the fuck is Tribit?” that you realize the disciplined and thoughtfully constructed replication between internal and external DNS servers may not be as robust as you thought.

The profanity belongs in the story. It is the point where a tidy model of replication meets the organizational surface of the failure. The operator is no longer looking only at configuration and process state. Email, web pages, other people’s work, and the name attached to the change have entered the same system.

What I checked was my change. The record I had edited resolved correctly. The servers had started. By every test I thought to run, the work was done.

The records that were blackholed were the ones below mine in the file. It had not occurred to me to be concerned about them, and nothing warned me. The scope of my verification matched the scope of my edit. The scope of the failure did not.

We restored the previous configuration from a backup within minutes. It did not matter. The bad answer was already cached in resolvers we did not operate. There was nothing left to fix and nothing to do but lower the TTL and wait for other people’s caches to forget.

I learned the hard way what low TTLs buy you: less time for an error to remain visible across the organization. That is not the same as preventing the error. It changes the duration of the consequence and the shape of recovery.

Mail started moving again later that afternoon. What told us was not the configuration file. We queried the authoritative servers directly, outside our own caches, watching for the corrected zone to appear carrying its new serial number, and then watched the MX records come back. The signal that mattered arrived from outside. It came from the same place the failure had: resolvers we did not operate.

The incident was forgotten soon enough. The lesson was not.

The point was not that a particular version of BIND behaved badly, nor that DNS configuration deserves special care, though it does. It was that component health and system health can diverge.

There was no dashboard to be green. That is worth being precise about, because it is the more common situation. The only signals we had were a process that started and a file transfer that succeeded, and both were accurate. What was missing was monitoring, configuration checking, and any safeguard standing between a routine edit and an entire company’s mail. A process can be present. Replication can be disciplined. The path another person depends on can still be broken.

Green is an observation. It is not absolution.

Certainty and the invisible operator

When troubleshooting, a bottom-up approach is often helpful, but certainty is the enemy of good analysis.

I have seen teams spend hours exploring paths they were certain must contain the failure, only to realize they could have picked up the phone and learned that their provider was experiencing an outage. The technical depth of the investigation did not compensate for the certainty that fixed its direction.

Sometimes digging in is the wrong thing to do.

That is difficult to recognize because operating rewards persistence. It rewards the willingness to look at the packets, the disks, and the text scrolling across the screen. But persistence can harden into attachment to a theory. During a transition, when familiar guarantees are already moving, the cost of that certainty increases.

The real work of an operator is to be invisible.

When the work succeeds, the transition completes and the system settles into a state other people can depend on. The knowledge, judgment, recovery paths, lost sleep, and discarded theories that made that possible recede from view. This is not a heroic story about the operator. Invisibility is the desired operating condition.

I am writing toward the person I was five years earlier, who might have mistaken a running process for a working system, and toward the person I will be five years later, who may have better names for the mechanisms I can only partly describe now.

The question I want both of them to keep asking is not whether everything can be replaced.

It is: what must endure while we replace it?