Imagine you take over a team that's been running without you for six months. There are runbooks. Good ones, actually — detailed, well-organized, someone clearly put effort in. You follow every step. Three months later the books are a mess. Not because you got the steps wrong. Because you didn't know why the steps existed.
One vendor had a quirk. Every invoice they sent had a duplicate line that their AP system auto-generated — the previous controller knew to strip it before posting. That knowledge wasn't in the runbook. It was in her head. You posted the duplicates. Six figures of phantom expenses, three months compounding before anyone noticed.
The runbook was a snapshot of what worked when conditions matched. The conditions changed. The runbook didn't know.
This is the actual failure mode of automation. Not the dramatic crash. The quiet drift. The process runs fine. The output looks plausible. And nobody notices until the audit.
The Supervisor Does Two Jobs. Automation Captures One.
Every supervisor is doing two things simultaneously. First: following the process. Second: noticing when the process doesn't fit the situation.
Automation is very good at the first job. That's what it was built for. Rules, steps, conditions, outputs. It executes precisely, without fatigue, at scale. If the process is right, automation is faster and cheaper than a human every time.
The second job is the problem. The moment the situation drifts outside what the process was designed for, someone has to notice and decide. That someone is the supervisor. And her ability to notice isn't in any document — because it can't be. Michael Polanyi named this in 1966: we can know more than we can tell. The tacit dimension. The supervisor knows things she learned by doing the work — pattern recognition accumulated over years, the feel of when something is off, the memory of the last time a number looked like this. None of that fits in a runbook. It wasn't documented because it couldn't be. It was the part of the job that lived in her, not on paper.
David Autor, the MIT economist, took Polanyi's idea and ran it through labor economics in a 2014 NBER paper. His point: automation reliably captures explicit, codifiable tasks and reliably fails to capture tacit ones. It's not that the engineers are lazy. It's a categorical limit. You can't automate judgment you can't articulate.
When you remove the supervisor and replace her with the process, you get the process. What you don't get is the part that knew when to override it.
August 1, 2012. 45 Minutes. $440 Million.
Knight Capital Group was the largest equity trader in the US. On the morning of August 1, 2012, they deployed new trading software. One engineer missed a step — failed to push the update to one of eight production servers. That server was still running old code.
The old code contained a function called Power Peg. It had been retired nine years earlier, in 2003. Nobody deleted it — they just stopped using it. At some point, the original developers who knew what Power Peg did had moved on. The feature flag that once activated it was repurposed for a different function in the new system. On that one unpatched server, the repurposed flag woke Power Peg back up.
Power Peg immediately started placing orders. Millions of them. The function had been written to execute trades in blocks — but the completion logic, the part that said "stop when done," had been removed years ago as part of the decommission. So it kept going. In 45 minutes, Knight Capital executed approximately 4 million trades across 154 stocks, moved 397 million shares, and lost $440 million. The company was dead within a year.
The SEC administrative order (Admin. Proc. File No. 3-15570) is worth reading if you want the full picture. What it documents isn't recklessness. Every individual step followed some logic. The deployment engineer followed the runbook. The system executed its instructions. The catastrophe wasn't a rule being broken. It was a rule being followed after the reason for the rule had expired.
The intent behind "don't activate Power Peg" lived in the people who built it. When they left, the intent left with them. What remained was code and a feature flag — and code doesn't know why it's there.
"Just Write Better Specs."
This is the first thing people say. Better documentation. Better prompts. Tighter rules. If intent went missing, that's a process failure — fix the process.
I'm convinced this response misunderstands what a supervisor actually does.
A supervisor doesn't succeed by applying rules correctly. She succeeds by noticing when the rules don't fit — and then inventing the right call in the moment. The rules are a starting point. The job is everything that happens when reality breaks the rules.
You can't spec that. Not completely. You can enumerate the edge cases you've already seen, which means you're protected against the last failure and exposed to the next one. Every automation project in history has hit this wall. A decade of RPA post-mortems will tell you the same thing: the pilots work, the enterprise rollouts underperform, and the gap is always the exception handling. The cases nobody thought to document because nobody had seen them yet. The supervisor handled those. The software doesn't.
Agentic AI will produce the same post-mortems in the same shape, for the same reason. The people selling you agents this quarter are promising you the first job — reliable execution of the process at scale. What they are not promising, and cannot promise, is the second job. The noticing. The override. The moment when someone has to say: this output looks right, but something's off, let me pull the thread.
You can't prompt your way around tacit knowledge. You can only push the gap to a different place.
The Chatbot Is the Company
In 2024, Air Canada deployed a customer service chatbot. A passenger named Jake Moffatt's grandmother had died. He asked the chatbot about the bereavement fare policy. The chatbot told him he could book at full price and apply for the bereavement discount retroactively, within 90 days.
That wasn't the policy. The actual policy required pre-booking approval. Moffatt booked based on the chatbot's guidance. Air Canada denied his retroactive claim.
When Moffatt took it to the BC Civil Resolution Tribunal, Air Canada argued — with a straight face — that the chatbot was "a separate legal entity responsible for its own actions." The tribunal rejected that argument in about one paragraph. Tribunal member Christopher Rivers wrote: "It makes no difference whether the information comes from a static page or a chatbot." Air Canada was liable. Moffatt got his $812.02 CAD.
This is the Air Canada case. February 14, 2024. It's already settled law in at least one jurisdiction: you cannot legally disown your own automation.
What's interesting isn't the legal outcome. It's the mechanism. A human Air Canada agent, applying judgment, would have known the policy and either applied it correctly or escalated. The chatbot knew the words of the policy. It didn't know the enforcement intent — that the policy existed to require approval before booking, not after. Knowing the words and knowing the intent are two different things. The chatbot got the words. The human supervisor would have gotten the intent. Mention isn't enforcement.
Air Canada removed the supervisor. The chatbot ran fine, technically. It answered questions. It cited information. It was helpful and prompt and never took a sick day. It just quietly promised something the policy explicitly didn't allow, to a grieving passenger, with no one in the room to catch it.
Three Questions for Any Agent Vendor
Here's what I'd ask anyone selling you agentic AI this quarter — especially for anything that touches your books, your customers, or your compliance obligations.
First: when the rules don't fit the situation, what does your agent do? If the answer is "it follows the rules," you're back to the Knight Capital problem. The rules are the starting point. What matters is the exception handling. Get specifics.
Second: who signs the output? Not who configured it. Who is accountable when it's wrong. If the answer is "the agent," or there's a long pause, treat that as a red flag and not a small one.
Third: how is judgment preserved when people leave? The vendor's team will turn over. The person who designed the rules for your specific GL structure, your specific vendor quirks, your specific definition of "that doesn't look right" — who holds that, and what happens when they're gone? This is the question that almost nobody asks, and it's the one that bites you three years in.
At Agentic CPA, the answer to all three is the same: the CPA stays in the loop. Our agents handle the execution — the processing, the categorization, the reconciliation, the prep work. The judgment — what correct means for this client, what a suspicious variance looks like in this industry, when to escalate versus absorb — that stays human. Not because I'm nostalgic about human labor. Because that's the part that doesn't generalize. It's context-specific, it's tacit, and it requires someone with skin in the game to hold it.
The pitch for fully autonomous agents assumes the supervisor is overhead. A cost center. The thing you remove to get to the software margin.
She isn't. She's the operating system. The rules are what she wrote down. The job is everything she knows that she couldn't write down.
Every automation project eventually finds the edge of the rules. What it finds there is either a human with judgment or a failure mode nobody documented. Knight Capital found the failure mode. Air Canada found it. A decade of enterprise automation projects found it, one quiet exception at a time.
The process runs fine without the supervisor. The books don't.
Someone still has to be in the room. Not to do the work — the agents can do the work. To know what "done right" means. That knowledge doesn't live in the runbook. It lives in the person who's been wrong about it before and remembers what it cost.