Blog

Thesis · AI agents in production

Agents don't fail on intelligence. They fail on rules nobody sealed.

The models are good enough. That is the uncomfortable part of the agent story in 2026. An agent can read a 40-page credit policy, pull a bureau report, check affordability and draft a decision memo in under a minute. Yet most agent projects never reach production. The agent isn't too dumb for the job. The problem is that nobody in the company can say, in one approved place, what the agent is allowed to do. Until someone can, nobody signs off.

So why do AI agents fail to reach production? In most enterprises the blocker is trust, not capability. An agent goes live when the people accountable for a process (the process owner, risk, compliance) can predict what it will do, control what it may do and prove what it did. That requires rules that are explicit, approved by an owner, versioned, and applied outside the model. In most companies today, the rules are none of those things.

The pilot worked. Then what?

The numbers are consistent. Camunda's 2026 survey of 1,150 senior IT and business leaders found that only 11% of agentic AI use cases reached production in the past year. Gartner projects that more than 40% of agentic AI projects will be canceled by the end of 2027. And when EMA surveyed 202 enterprise IT and security leaders for Cequence Security in September 2026, 65% said an AI agent had already acted outside its intended scope.

None of those numbers say the models failed. They describe projects that worked in a demo and stopped somewhere after it.

A head of AI transformation at a financial firm described it to us precisely. You can get a working demo up in no time. Then you spend 99% of the time on the last 1%.

We have seen what that last 1% looks like many times, and it is usually a meeting. The pilot has run. The numbers look good. Then someone from risk, or the process owner, or internal audit asks three questions:

  1. Which rules does this agent follow, exactly?
  2. When those rules change, how does the agent find out?
  3. If an examiner asks next March why it approved this case, what do we show them?

The room goes quiet, because the honest answers are "the ones in the prompt, as the model reads them", "someone updates the prompt" and "the logs, probably". Projects rarely die in a postmortem. They die in a calendar where the next meeting never gets booked.

Where the rules actually live

Ask any operations leader where the rules for a process live and you will get a list. The credit policy. The underwriting manual. Two procedures, one of them from 2024. A regulatory update someone summarized in an email. And a senior underwriter who has been there for 20 years and knows which of those documents to ignore.

We hear versions of this on almost every call. A risk executive at a US bank told us that the rules for ACH were spread across eight different procedures, and nobody had the full picture end to end. An executive at a credit union described ten people in the same department doing the same job, each one doing it differently, each one sure they were right.

People cope with this. They absorb the contradictions through experience and quiet conventions ("policy beats procedure", "ask Sarah"). An agent can't do that. It reads what it is given and does its best.

Here is what that looks like. Say the credit policy, section 4.1, version 3.2, sets a maximum loan-to-value of 80%. The lending procedure, section 2.3, last updated in 2024, says 85%. A senior underwriter knows the policy wins. An application arrives at 83%.

The agent retrieves both passages. It approves or declines depending on which one carries more weight in that particular run, how the prompt was phrased, and which model version is live. Tomorrow, on an identical application, it may decide the other way. Nobody chose 80% or 85%. The agent did, by accident, and nobody can tell you which way it went until someone looks.

This isn't a hallucination. The agent faithfully read two documents that disagree. The failure sits upstream of the model, in the rules themselves.

Scale changes what this costs. One underwriter misreading a threshold affects a dozen files before someone catches it. An agent applies the same ambiguity to every case it touches, thousands a month, at a speed no review team can match. Ambiguity that people had been quietly absorbing for years becomes a production incident. We priced that out in What does an AI agent mistake cost?

There is also the question of ownership. In most agent pilots, the rules end up in a system prompt, and the system prompt belongs to whoever built the agent. When the credit policy changes, the team behind one agent updates its prompt. The team behind another agent finds out a month later, or never. The Head of Credit, who is accountable for the rule, has never seen either prompt.

A capable model reading unclear rules is still guessing

A language model is probabilistic by design. Ask it the same question a thousand times and you get a distribution of answers, most of them good. That makes it excellent at reading a messy document, summarizing a SOC 2 report or drafting a memo. It also means that when a model holds the rule, every decision is a fresh interpretation.

The usual fixes don't change that. Setting temperature to zero makes the output less variable, but the model is still reading your policy and deciding what it means. A model update can change that reading overnight. A longer prompt adds more text to interpret. Retrieval finds the relevant passages, including both of the ones that disagree.

Put probabilistic reasoning on top of contradictory rules and you get decisions nobody can reproduce or explain. That is the exact thing risk and audit are paid to stop.

This is also why accuracy is the wrong scoreboard. A team reports that its agent is 98% accurate on a test set. Risk asks which 2% it gets wrong, and whether anyone could have known in advance. In regulated work, and in any process where one error is expensive, an average doesn't help. The question is whether you can show that this specific decision followed this specific rule. A capable model that holds the rule can't answer that, however good it is.

The tooling most teams add next is useful, and it attacks the problem from the wrong end. Guardrails filter what an agent says. Evaluation suites and model-as-judge setups score outputs after the fact, and their bill grows with every loop. Observability records what happened. Each of them works on the model's output after the model has already interpreted the rule. They make the guess better, or catch some bad guesses. The guess itself is still there. We compare all six common approaches in Six ways to keep an AI agent on policy.

EMA's August 2026 research puts a number on it: only 34% of enterprises check an agent's authority at the moment it acts. Everyone else finds out afterward, if at all.

A system prompt is not a Rulebook.

The fix: take the rule away from the model

The thesis behind Factify is simple to state. Take the rules out of prompts and people's heads, settle them once with the person accountable for them, and make every agent run on that one version. The model keeps doing what it is good at, which is language: reading, extracting and drafting. It stops holding the rule.

We do it in three steps, in a fixed order.

Each step removes one way an agent decision fails1 · Sealone approved, versioned truthremovestwo documents,two answers2 · Enforcechecked outside the modelremovesthe model decideswhat the rule means3 · Proveevery case on the recordremovesnobody can showwhich rule applied
Seal, then enforce, then prove: each step removes one failure, and each depends on the step before it.
1 · Seal

Rulebook Studio pulls the rules out of your policy, procedures, regulation and tribal knowledge, and flags every conflict and contradiction it finds. The domain owner, here the Head of Credit, reconciles them. The 80% versus 85% question gets decided once, by the person accountable for it, with a name and a date on the decision. The result is one Sealed Truth, versioned and owned by you.

2 · Enforce

The Sealed Truth is compiled into an Executable Factified Rulebook, which agents use as tools through MCP or API. At each step the agent asks the Rulebook what to do next, does the work, and hands the result back for checking. Nothing leaves a step unless it matches. The model still reads the application and drafts the memo. It no longer decides what the loan-to-value limit is. Same case, same decision.

3 · Prove

Every case lands in the Ledger, with the Rulebook version it ran on, the inputs, and every judgment that was made along the way. When the examiner asks about a case from March, you replay it and show which rule applied, who sealed it and what the agent did at each step.

The order matters more than it looks. If you enforce rules that were never sealed, you enforce the contradiction faster. If you prove without enforcing, you get a detailed record of what went wrong. Most agent stacks today start at the end, with logs and observability, or in the middle, with guardrails. Very few start where the problem is, with the rules themselves.

There is a second-order benefit that AI leaders tend to notice first. Once the rule lives outside the agents, it stops being copied into each of them. One Rulebook, every agent. Change a rule once, and every agent follows it from the next case. It doesn't matter whether that agent was built in-house or on a vendor platform, or which model it uses.

Three fair objections

When we lay this out for banks and AI teams, three objections come up. They are worth answering directly, because each one is partly right.

"Isn't this just a rules engine? The lower end of AI." A CISO at a community bank said almost exactly this to us. Rules engines run logic that someone in IT wrote by hand, and every change goes through a release cycle. The business can't read the rules and doesn't own them. A Sealed Truth starts from the documents and tribal knowledge you already have. The domain owner, not a developer, decides what the rule is. And the model stays in the loop for everything a rules engine could never do: reading a bank statement, extracting the findings from a SOC 2 report, drafting the memo. The model is still there. It just isn't the authority.

"Our work runs on judgment. Rigid rules will kill what makes us good." An AI program lead at a bank told us that part of their edge in commercial lending is making the deal work that wouldn't get done on a grid. He's right, and it's an argument for deciding where determinism stops, not for skipping it. Look at the steps of a loan: verify identity, pull the credit report, check affordability, decide whether the risk is reasonable, then price and offer. Three of those five steps involve no judgment at all. Identity verifies or it doesn't. Debt-to-income is a number. The fourth step is where a credit officer earns their salary. In a Factified process, the Head of Credit marks it as a judgment point and names the judge: a person, an agent with the authority to decide, or a person above a threshold and an agent below it. Judges step in at owner-defined points, or when a case isn't covered, and their decision is recorded with the case. Deterministic where you choose. Judgment where you need it.

"Can you guarantee the agent won't make mistakes?" No, and nobody honest can say that about a model. What we bound is what the agent is allowed to do. No improvising. No off-Rulebook decisions. Uncovered cases go to a judge, and every case is on the record. Whether the Rulebook itself meets a regulation is a call for your owners, and the Ledger gives them the evidence to make it.

What this is not

This isn't a bet against models. Better models make the approach more valuable, because once the rule sits outside the model you can hand the model more of the work with less risk.

It isn't a replacement for your agent platform either. Most companies we talk to run more than one: a vendor copilot, a CRM's agents, something built in-house. Each governs rules its own way. A Rulebook sits under all of them. Any agent, any model, any platform plugs into the same truth.

And it isn't a reason to slow down. Most teams spend their time on the last 1% because they are reverse-engineering rules from model behavior, one failed test at a time. Sealing the rules first makes that work shorter.

What to do on Monday

You can test this thesis on your own process in an afternoon, without buying anything.

  1. Pick one process that is close to production, or that you would put agents on first if sign-off weren't the problem. Payables, vendor due diligence and account maintenance are good candidates.
  2. Count the documents that define it. Write down the number. Then write down the names of the people who know rules that aren't in any of them.
  3. Check the three numbers that matter most. Take the thresholds, limits and approval levels the process depends on, and look each one up in every document. Each disagreement you find is a decision your agent would otherwise make for you.
  4. Name the owner for each conflict. Not the team. The person. If nobody can be named, you have found the real reason the project is stuck.
  5. Ask the audit question now. If someone asked why the agent decided a specific case last month, what would you show them? If the answer is "logs" or "we'd rerun it", you can't prove it yet.

If the answers are clean, you're ready to agentify. If they aren't, the order is clear: Factify it, then agentify it.

Bring one process. See it Factified. A working session with our team on one of your real processes: where its rules live today, where they conflict, and what it looks like sealed.

Book a session

Frequently asked questions

Why do most AI agent pilots fail to reach production?

Most pilots stall on trust, not capability. The people accountable for a process can't predict, control or prove what the agent will do, because the rules it follows sit in prompts, scattered documents and people's heads. Camunda found only 11% of agentic AI use cases reached production in the past year.

Will a better model fix this?

A better model reads documents more accurately, but it still interprets the rules on every run. If two documents disagree, a better model reads both of them better. Consistent decisions come from rules that are settled by an owner and applied outside the model.

What is a Sealed Truth?

A Sealed Truth is the single, approved, versioned set of rules for a process. It is extracted from policy, procedures, regulation and tribal knowledge, with conflicts resolved by the domain owner. Factify compiles it into an Executable Factified Rulebook that agents call as tools.

Does this replace our agent platform?

No. The Rulebook sits under your agents, through MCP or API. Agents built on any platform or model run on the same truth, so a rule changed once applies to every agent from the next case.

Where does human judgment fit?

At the steps the domain owner marks as judgment points, and at any case the Rulebook doesn't cover. A judge, either a person or an agent with named authority, decides there, and the decision is recorded with the case in the Ledger.

Bring one process. See it Factified.

A working session with our team on one of your real processes: where its rules live today, where they conflict, and what it looks like sealed.