An air traffic controller watching a busy scope will, sooner or later, miss the one aircraft that matters. Sustained attention decays under load, a limit aviation named the vigilance decrement and has spent seventy years designing around.
AI has moved every knowledge worker into that chair. You now work a dozen aircraft at once: six open conversations, an agent drafting in the background, three more waiting, and someone calls it efficiency. Your scope is your inbox. The vigilance decrement does not care that you were told to keep up.
What aviation did about it
Aviation did not answer the vigilance decrement by telling controllers to concentrate harder. It built a machine that watches on its own, then let the machine overrule the human.
Aviation’s response was a rule that still governs every cockpit: when the machine and the human disagree, follow the machine. And note what kind of machine it was. A collision-avoidance system reads no intent; it computes two converging tracks and calls it. Aviation did not teach crews to trust a clever machine. It taught them to obey a dumb, certain one.
Security made the opposite bet
Security bet the other way. For 30 years, its last line of defense has been a person noticing that a message is wrong. AI just made that a losing bet twice over.
First, it erased the tell. In one controlled study, Bruce Schneier and Fredrik Heiding found AI can cut the cost of a phishing campaign by more than 95 percent while matching the click-through of a skilled human team. The clumsy, typo-ridden lure we taught people to catch is disappearing.
Second, AI erased attention by sitting every reader in the controller’s chair. Switching degrades attention, so we are asking a saturated controller to catch the one hostile aircraft on a scope that aviation already proved they will miss.
Our problem is harder. A converging jet is not trying to hide; it shows on the scope as exactly what it is. A fraudulent wire is built to look legitimate, and the adversary can pick the moment you are most swamped. Aviation’s machine had to out-watch human fatigue. Ours has to out-think an opponent.
Watch the ones that can kill you
You cannot watch every aircraft equally, so watch the ones that can kill you, and the threat data says which they are.
The FBI recorded 21,442 business email compromise complaints in 2024, out of 859,532 total complaints, yet BEC accounted for $2.77 billion of the year’s $16.6 billion in reported losses. These attacks represent a small fraction of incidents but a disproportionate share of the damage because they lead to consequential actions: a wire, a change of bank details, a credential reset. Rare and hard to reverse.
This is where aviation’s rule actually lands, and it lands on the dumb machine. Guard those actions with something that fires on the event itself, not on a human’s read of it: a mandatory callback to a known number for any change of bank details, dual control on a wire, a hard threshold above which a payment stops on its own.
None of this is new. These controls used to serve as belt and suspenders behind an email filter that caught the obvious lure. The tell is disappearing, so these controls have to carry more of the load. Like the collision system, they read no one’s intent. They fire on the act, every time, and ask no one to be clever.
What they miss is the wire that is already coherent: a request from a genuinely compromised vendor thread, correct in every detail, that a callback confirms because the fraudster holds the other end too. Catching that means judging whether an action makes sense in context, the adversarial problem a collision system never had to solve.
This is where aviation’s rule stops transferring, at least for now. You cannot tell anyone to follow this machine because it is not the dumb, certain kind. It guesses in an adversarial fog, and a wrong guess halts a real wire. Any machine in the money path earns its place only if it is quiet enough to be trusted and cheap enough when wrong that treasury does not route around it.
A noisy machine just relocates the vigilance decrement onto the backup, who clears its alerts on the same schedule the controller missed the blips. The reliability required to ever say “follow the machine” is the problem security still has to solve.
The question worth ending on
Aviation reformed because the cost of pretending was a wreck in a field with 71 names on it. Security’s cost is quieter: a wire that clears and an identity handed over, so we keep asking people to do the one thing aviation proved they cannot: serve as the reliable last line against a rare event buried in routine. AI has made that bet even worse.
The discipline that learned this first did not demand more vigilance. It built controls around the moments where failure carried the greatest consequence and stopped depending on an overloaded person to catch every dangerous event.
Security already has its version of the dumb machine: controls that fire on the act, regardless of whether anyone recognizes the attack. The machine that reads intent has to earn that same trust before anyone follows it.
So the question is not whether your people are paying attention. Aviation could have told you decades ago that, eventually, they will not. The question is whether, at the moment someone is about to move money or surrender identity, anything stands in the path except a human hoping to notice.
No Responses