Why most enterprise AI agents fail in week six — and the operating model that fixes it
The pattern
It happens with uncanny consistency. The pilot ships in two weeks. The demo lands. Leadership is energized. Then, somewhere between week five and eight, the agent starts degrading in production. Exception queues build. The ops team loses trust. The project goes on hold. Everyone learns a lesson nobody can fully articulate.
I have watched this play out inside five different enterprise clients in the past year alone. The cause is never the model — Claude, GPT-4, Gemini all perform well in demo conditions. The cause is almost always the same two missing ingredients: evals-first development and a clear human-in-the-loop contract.
The demo problem
Enterprise AI demos are optimized for the happy path. You assemble the fifteen most representative examples, tune the prompt until they all pass, and put that in front of leadership. What you have not done is model the long tail — the edge cases, upstream data quality failures, ambiguous instructions, and system states your test set never touched.
Without an evaluation suite that mirrors production variance, you will not know you have a problem until you are already in production. By then you have burned the stakeholder trust that took months to build.
The agent does not degrade. Your understanding of it degrades — because you were never testing against reality.
What evals-first looks like
Before a single agent is written, I define the evaluation contract: what inputs the system will encounter, what good outputs look like, how we score them, and what failure rate triggers a rollback. This is not unit testing — it is a structured sample of real production conditions, including deliberate edge cases and known failure modes.
The eval suite runs on every prompt change, every model swap, every deployment. The 94.7% auto-resolution rate you see in my case studies is not luck. It is the output of knowing, precisely, what the agent is being asked to do — and having continuous signal on whether it is still doing it well.
The human-in-the-loop contract
The second failure mode is scope ambiguity. When an agent is tasked with "handle customer service," nobody has defined what the agent owns versus what escalates to a human. In week one, operators watch closely and catch edge cases by hand. By week six, they have stepped back. The agent is handling decisions it should not, and the ops team has no clear protocol for taking back control.
A good human-in-the-loop contract defines three things: the confidence threshold below which the agent self-escalates, the exception types that always route to a human regardless of confidence, and the override mechanism that lets the ops team reclaim any decision in under 60 seconds.
The six-step playbook
- Ops audit first. Before touching the model, map the actual workflow — exceptions, edge cases, what the human team actually spends time on. This is where your eval cases come from.
- Define the operating envelope. Not "handle everything" — "handle these request types, under these conditions, above this confidence threshold." Everything outside escalates.
- Build evals before agents. The eval suite is the contract. Write it first so every subsequent build decision can be scored against it.
- Week-one shadow mode. The agent runs in parallel with the existing process, no live decisions. You are building trust data and surfacing edge cases before anyone depends on the output.
- Graduated authority. Low-stakes decisions first. High-confidence outputs first. Expand scope only when eval scores justify it.
- Override by default. Every agent decision should be reversible in one click — not as a fallback, as a design principle.
The pilots that survive week six are not running better AI. They are running better operating models. The model is the easy part.