Runtime Is the Moment of Consequence in AI

Correction note (2 October 2026): This article now bounds its claims to documented incident records and specified agent architectures, removes unsupported market-wide language and regulatory detail, narrows the approval-fatigue evidence to three load-bearing findings, and reports the Therac-25 control-room failure without worker-identifying detail. Earlier text also used an unsupported historical superlative and overstated what deterministic policy can guarantee.

In March 1986, during a radiation treatment at the East Texas Cancer Center in Tyler, Texas, a rapid correction from X-ray mode to electron mode left the Therac-25 in an unsafe state and the beam fired. The patient felt what was later described as an intense electric shock and began to rise from the table. The console showed none of this. It displayed MALFUNCTION 54, a code the manual did not explain, over a dose reading close to zero. A Proceed command followed, and the beam fired again as the patient was getting up. The intercom was broken that day and the video monitor was off, so the control room had no working audiovisual view of what was happening inside. Each firing delivered roughly one hundred times the prescribed radiation dose. The patient died of radiation-induced myelitis five months later.

Three weeks later, the same machine delivered another overdose during a treatment. That patient also died. The Therac-25 investigation documents six known overdoses between 1985 and 1987. It does not support the historical superlative this article previously used.¹

Nobody designed the Therac-25 to kill. It killed through three failures working together:

• a control room conditioned by repeated false alarms,
• training that said overdose was impossible,
• and no effective independent hardware interlock preventing that firing state.

The failure was not in the intention, and not in the design review. The failure was in the moment the machine acted.

This piece is about the moment a proposed action becomes a real effect: who stands in it today, what it does to them, and what we owe them instead.

AI agents now act in the real world: they can move money, change account credentials, write into records and operate infrastructure. In high-stakes environments the unit of harm is often one action, not one session. Some agent deployments rely on a human approval prompt as the final control without a separately enforced execution boundary. That is the architecture examined here. The claim is not that every deployment has this shape.

Software was the modern choice

The Therac-25 was built by AECL, the Canadian crown corporation that designed the CANDU reactor, as its flagship move into medical infrastructure: a dual-mode radiation therapy machine, electron beam or X-ray from a single unit. One machine instead of two. One operator console instead of two, a real revolution at the time. By 1985 there were eleven of them in hospitals across Canada and the United States.

One design decision mattered more than any other. The older Therac-20 used hardware interlocks: physical switches that made it mechanically impossible to fire the high-current beam without the X-ray target in its path. The target is a small metal plate that converts the machine’s raw electron beam into the gentler X-rays used for treatment; without it, the patient receives the raw beam at full power. The Therac-25 replaced those switches with software checks. Software was newer, cheaper, more flexible. Software was the modern choice.

Unbeknownst to anyone, the Therac-20’s hardware interlocks had been masking software defects for years. The Therac-25 shipped without them.

Six accidents followed between 1985 and 1987. Patients were severely injured, and at least three died.

The accidents did not have one technical cause. Leveson and Turner documented race conditions, unsafe software design, reused code, weak incident investigation, confusing interfaces and the removal of hardware interlocks. Fast editing exposed one path to failure. Treating the timing flaw as the whole explanation would repeat the design mistake: looking at one component instead of the system.¹

Three burdens sat on the people in that control room, not one.

The first was structural. No effective independent hardware interlock prevented the high-current beam from firing while the target plate was absent. The hardware interlocks that would have blocked that firing state had been removed, because software was deemed enough.

The second was fatigue. The error messages were cryptic: the word MALFUNCTION and a number from 1 to 64. The same warning pattern covered every level of risk, from benign to lethal, with nothing to distinguish the two. An FDA memo written after one of the accidents found that the operator’s manual “does not explain nor even address the malfunction codes,” and that the materials “give no indication that these malfunctions could place a patient at risk.”¹ One clinic reported up to forty dose-rate malfunctions on some days, while training presented an overdose as effectively impossible.¹ A cryptic error code followed by a Proceed key is exactly the shape of an Always Allow button. The pattern has been live since at least 1986.

The third came after the harm. After the second Tyler overdose, hospital staff spent a weekend investigating the machine because AECL refused to accept that the bug existed. They found that fast editing on the data-entry screen triggered the malfunction and reproduced it at will. The hospital, not the manufacturer, had to prove the bug existed in production after patients had already been harmed.

Following the accidents, Nancy Leveson and Clark Turner spent three years assembling the canonical investigation.¹ One finding anchors this entire article:

The investigation found that commonplace malfunction messages, most unrelated to patient safety, had made the warnings lose their force. The system produced false alarms, trained the control room to trust it, and then left a human as the last line of defense.

Later medical-device standards formalized software lifecycle processes, but no standard can be credited with “solving” Therac-25. The durable lesson is narrower: software assurance, hazard analysis, incident evidence and independent safety mechanisms have to be designed as a system.

The same shape, three times

The Therac-25 is not an outlier, but the events below are analogies, not one historical law. Their mechanisms and regulatory histories differ.

In 2018 and 2019, two Boeing 737 MAX aircraft crashed, killing 346 people. Official reviews describe unintended MCAS operation, certification assumptions, cockpit alerts and inadequate system-level analysis. Reducing that record to one sensor or one piece of software would be misleading. Congress later enacted the Aircraft Certification, Safety, and Accountability Act.²

In 2012, Knight Capital’s deployment errors activated obsolete Power Peg code. The SEC found that the resulting orders caused losses of more than $460 million in approximately 45 minutes. It also found deficient controls and monitoring under the existing Market Access Rule. The enforcement record does not support the article’s earlier claims about a 17-minute diagnosis or a new rule written in response.³

The shapes were not identical. A radiation machine fired its raw beam with the target plate out of place. An aircraft fought its pilots after erroneous sensor input. A trading system reactivated dead code. Across these different systems, design-time approval did not determine what happened at execution. The runtime layer is where harm goes from hypothetical to real.

The shared point is modest. Design-time approval does not tell you what a deployed system will do in every consequential moment.

Where AI agents are right now

Frameworks from NIST and threat catalogs from OWASP and MITRE address parts of the problem. Their scopes are specific. They can guide risk management and threat analysis, but a document or catalog does not itself sit in an application’s action path. The companion governance article examines that documentation-to-execution distinction. This article stays with the narrower safety question: what control is present when a consequential action is attempted?

Between April 17 and early June 2026, Meta notified 20,225 people that attackers had used its AI support experience in account-takeover attempts.⁴ The public filing says a separate code path failed to verify the supplied email. It does not establish the article’s earlier story that the model understood the request and called the right tool. The supported point is simpler: a conversational surface reached a consequential account-recovery path without the required verification.

Here is the gap, plainly. Guidance and product frameworks can describe or implement pre-execution controls. NVIDIA NeMo, for example, documents execution rails that can inspect or block tool calls.⁵ The control still works only when the application routes the consequential action through it. What is the equivalent, for an AI agent that moves money or medical records, of never fire the beam without the target plate in place? The rule has to be explicit, versioned and enforced outside the model’s generated reasoning.

In an application that has no such boundary, the fallback may be a human approval prompt. That is a design choice, not a universal description of agent deployments.

How many times did the AI ask you to click Allow

A question:

How many times has Claude or ChatGPT asked you to approve something, and you clicked Allow without reading the question?

Many users will recognize the pattern. In one experiment using a fictional social network, ninety-eight percent of participants missed clauses that included giving up their first-born child.⁶ A separate experiment found that neural responses to repeated security warnings declined with repetition. It did not show that the visual cortex stopped processing them entirely.⁷

Now look at where the same pattern can land in production. A BIS Innovation Hub report summarized industry estimates that 90 to 95 percent of generated anti-money-laundering alerts were false positives. That is an attributed estimate, not a universal measured rate.⁸

A human approval step is only as useful as the signal, context, authority and time it gives the reviewer. Repeated low-value prompts can train people to click through. That is an analogy to the Therac-25 record, not a claim that the two systems share one design.

The human loop owes the reviewer signal, context, authority and time.

What action-level runtime governance is

Action-level runtime governance sits between an AI agent’s proposal and a consequential action. Before the application calls an API, writes to a database or executes a transaction, the boundary evaluates explicit facts about the request and policy. It then allows, denies or escalates the action, and records enough context to reconstruct what it observed and decided.

The deterministic filter is small and precise by design: a handful of catastrophic invariants, not a universe of correctness rules. The interlock did not make the older Therac correct. It made it physically unable to fire the raw beam. Same logic, new technology.

Deterministic policy evaluation is useful because the same normalized inputs, policy version and relevant state produce the same policy result. It does not prove that the policy is correct. A bad threshold or incomplete rule can still produce false positives and false negatives.

This can reduce review noise when the routing policy is well designed and measured. It cannot guarantee that a reviewer sees only the right actions. Teams still need to test escalation rates, misses, overrides and unevaluable cases. The deterministic rules hold a defined floor whether or not anyone is watching the screen.

The cost is application-specific. Policy evaluation, state lookup, logging and escalation all add latency and engineering work. Measure the complete action path before making performance claims.

A useful implementation starts with three things: a small set of explicit invariants, a guaranteed invocation point before consequential effects, and an evidence record that distinguishes proposal, authorization, execution and observed result.

For anyone running agents in production now, one question remains: what must your agent never do?

fivedrisk

While building DotOS, my personal AI systems lab, I ran into the same problem. I could not leave my AI agents running on their own with just probabilistic governance. So I started developing fivedrisk for my own projects, and published it in case the architecture is useful to others working on the same problem.

Fivedrisk is an Apache-2.0 open-source implementation of an action-level runtime gate with deterministic scoring along five dimensions: Data Sensitivity, Tool Privilege, Reversibility, External Impact, Autonomy Context. The public repository shows the code, decision records and current limitations. It does not establish regulatory compliance.

Closing

The histories are different. The engineering lesson is shared: a consequential system needs controls and evidence at the point where a proposed action becomes a real effect.

AI agents are in production now. The relevant test is whether the action path enforces the few boundaries that cannot be left to generated judgment. Runtime controls do not solve every failure.

What stops your agent at the action boundary when its reviewer has learned to click Proceed?

References