Complex systems almost never fail because one thing broke — they fail because a dozen small compromises finally lined up on the same bad day.
I want to spend some time on that idea, because it’s the single most useful mental model I know for understanding AI agents. If you’re new to agents, you’ve probably been told they fail when the model “hallucinates” or when someone writes a bad prompt. That’s a comforting story. It’s also mostly wrong, and believing it makes you worse at spotting real trouble.
Failure is a group project
Research on complex systems keeps landing on the same finding: failures cascade. One component degrades, something downstream compensates, that compensation stresses a third thing, and eventually the whole arrangement tips over. No single culprit. Just a chain.
Agents are a textbook case. A typical setup involves a language model, a set of tools it can call, an API or three, some memory or storage, a permission layer, and a human somewhere in the loop. Each piece works. Each piece has been tested. And each piece has quiet assumptions about what the others will do.
So when your agent does something baffling — books the wrong meeting, emails the wrong list, loops forever on a task — the honest answer is rarely “the AI got confused.” More often it’s that a tool returned an unexpected format, the agent improvised around it, the improvisation passed a permission check that was written for a different scenario, and nobody was watching that particular corner.
The light bulb problem
There’s a line from the literature on this that I think about constantly: maintenance and production are usually at cross purposes. If you shut everything down every time you have to change a light bulb, you’re never going to get anything done.
This is the tension at the center of every system that matters. You need it running. You also need to fix it. Those two needs pull in opposite directions, and production almost always wins, because production is what people notice.
Anyone who’s run software knows the shape of this. The update is available. The update requires a restart. The restart requires a maintenance window. The maintenance window requires telling users. So the update waits. Then it waits again. Meanwhile, digital threats are evolving fast, and many of those pending updates contain patches for security vulnerabilities — often aimed at defending against attack vectors that didn’t exist last quarter.
The gap between “patch exists” and “patch installed” is where a lot of failures live.
Why agents make this harder
Agents raise the stakes in a specific way. Traditional software waits for instructions. An agent acts on your behalf, which means a small malfunction doesn’t just sit there producing a wrong answer — it produces a wrong action, and then another one built on the first.
A few things that make agent systems particularly good at cascading:
- They’re chatty. More API calls and tool connections means more places for a mismatch to start.
- They improvise. Adaptability is the selling point, but it also means an agent will route around a broken component instead of stopping and telling you it’s broken.
- They compound. Step three inherits whatever step two got wrong, and nothing in the chain necessarily flags it.
- They hold permissions. Access to email, calendars, files, or payments turns a small bug into a real-world consequence.
Perspectives on this research make a related point that stuck with me: every new feature adds new failure paths to a system that was previously well understood. Each capability you bolt onto an agent isn’t just an addition. It’s a new way for things to go sideways, and it interacts with everything already there.
What non-technical people should actually do
You don’t need to become an engineer. You need to stop treating your agent tools as appliances.
Install the updates, and don’t put them off because the timing is inconvenient. Ask what an agent can actually reach — a tool with access to your inbox deserves more scrutiny than one that summarizes documents. Notice weird behavior instead of working around it, because working around it is exactly the compensation pattern that hides a cascade until it’s large. And keep a rough sense of how many services your agent touches, because that number is your real exposure.
Heading into 2026, the theme worth internalizing is proactive maintenance — fixing things before they announce themselves, not after.
Complex systems fail smoothly. That’s what makes them dangerous. Nothing screams before it breaks; the warning signs look like minor annoyances until they don’t. The people who stay ahead of that aren’t smarter about AI. They’re just more willing to change the light bulb.
🕒 Published: