Every team that puts an AI agent on a real process ends up asking the same question: how do we keep it on policy? There are six common answers, and most teams use three or four of them at once. Each one solves part of the problem. This post looks at all six against the same test, says plainly where each one is the right choice, and where it stops being enough. We sell one of the six, so we've tried to be fair to the other five.
The test every approach has to pass
AI guardrails, prompts, evaluations and the rest are all ways of answering one question: when this agent makes a decision, will it follow our rules, and can we show that it did? To compare them, it helps to break that into five criteria.
- Consistency. Does the same case get the same decision every time, whatever the model generates on that run?
- Ownership. Can the person accountable for the rule (the Head of Credit, the Head of Claims, the Controller) read it and change it, without filing a ticket?
- Change cost. When a rule changes, how long until every agent that uses it follows the new version?
- Proof. Six months later, can you show which rule, and which version of it, governed a specific decision?
- Cost to run. What does each decision cost in compute, latency and people?
We'll run every approach against the same case. A payables agent matches supplier invoices to purchase orders and releases payment. The finance policy says any invoice over $25,000 needs the Controller's approval before it is paid. An invoice arrives for $48,000. It matches its purchase order line for line.
1. Prompting, with retrieval over your policies
What it is. The rules go into the system prompt, and the agent can search your policy documents when it needs more. It is how almost every pilot starts, for good reason: it is fast to build, and models are very good at reading policy text.
Where it wins. Exploration, drafting, internal assistants and any task where a wrong answer is cheap and easy to spot. Retrieval is also the right way to give an agent access to long, changing documents it needs to read.
Where it breaks. The prompt is guidance, not a check. On our payables case, the agent will usually notice that $48,000 is over the limit and route the invoice to the Controller. Usually. Whether it does depends on how the prompt was phrased, which passages the search returned and which model version is live. A risk lead at a bank described their in-house policy search tool to us as "a glorified search function," and meant it fairly: it finds the right text, and then a model decides what the text means, fresh on every run. Ownership sits with whoever wrote the prompt. Proof is a log of what the model said.
2. Guardrails and output filters
What it is. A layer around the model that checks inputs and outputs: blocking unsafe content, personal data or prompt injection, enforcing a format, keeping the agent on topic.
Where it wins. Safety, privacy and format, and they are fast and cheap to run. Every production agent should have them, and none of what follows replaces them.
Where it breaks. AI guardrails check what an agent says. They don't know your approval matrix, and they aren't designed to. A guardrail can make sure the payment instruction is valid JSON and carries no personal data. It can't tell you that a $48,000 invoice needs the Controller's signature, unless someone writes that business rule into it, at which point it has become a small, hard-to-read rules system that IT owns.
3. Evaluations, model-as-judge and red teaming
What it is. Test sets of past cases with known right answers, a second model that scores the first one's output, and adversarial testing before release. Increasingly, the judge model also runs in production, checking each output before it is used.
Where it wins. Measuring quality before launch and catching regressions when a model or prompt changes. A good evaluation set is one of the most useful things an agent team can build, and it is also how you discover which rules the agent keeps getting wrong.
Where it breaks. An evaluation tells you how often the agent is right on the cases you tested. It doesn't make the next case right. A judge model in production is a second probabilistic reading of the same rule, so it catches some errors and adds its own. The cost also compounds: every generate, judge, regenerate loop multiplies the inference bill. An AI lead at a bank described those loops to us in one line: they burn through tokens. A fixed rule gives the same answer for the price of a lookup.
4. Observability and monitoring
What it is. Traces of every step the agent took, dashboards, drift detection and alerts when behavior changes.
Where it wins. You can't run an agent in production without it. It is how you see what happened, debug failures and notice when something shifts. For reversible, high-volume, low-stakes decisions, good monitoring is often the right control on its own.
Where it breaks. Monitoring tells you what the agent did, not whether it was allowed to. It runs after the action. On our payables case, a dashboard will show you, tomorrow, that $48,000 went out without the Controller's approval. By then the supplier has it. We went into this distinction at length in A dashboard is not a gate.
5. Hard-coding the rules, or a business rules engine
What it is. The business logic is written as code, or configured in a rules engine, and the agent calls it. The model handles language; the code handles decisions.
Where it wins. Consistency and proof. Code gives the same answer every time, and if you log the inputs and the code version, you can reproduce any decision. For a small number of stable, well-understood rules, this is a perfectly good answer, and plenty of teams ship it.
Where it breaks. Ownership and change cost. The rule now lives in a repository that the Controller or the compliance officer can't read. Every change goes through a development cycle. The code also only contains the rules someone already knew to write down. In most processes we look at, a large share of the real rules sit in procedures, emails and people's heads, and they conflict. A rules engine runs whatever logic it is given. It doesn't help you find the logic, or settle which of two documents is right.
6. A sealed, executable rulebook
What it is. The rules for a process are extracted from the documents and knowledge you already have, conflicts are surfaced, and the person accountable for the process settles them and approves one versioned rulebook. Agents call that rulebook as a tool at each step. Cases it doesn't cover go to a named person. Every case is recorded with the rule version it ran on. This is the approach we build at Factify.
Where it wins. All five criteria at once, for the decisions that matter. Same case, same decision. The domain owner reads and changes the rule. A change reaches every agent from the next case, whatever platform the agent runs on. And every decision can be traced to the rule and version behind it. On our payables case, the rulebook doesn't interpret anything: the invoice is over $25,000, so it goes to the Controller, every time, and the record shows which rule sent it there.
Where it breaks. It needs discovery work up front. Someone has to find the rules and the owner has to make decisions, and for a process with years of accumulated exceptions that takes real time. A deployment lead at an AI agent company told us that about two thirds of a typical project is discovery. A sealed rulebook doesn't make that work disappear; it makes it the explicit first step, done once, with tooling. It is also overkill for low-stakes tasks. You don't need a sealed rulebook for an agent that drafts meeting notes.
The comparison in one table
| Consistency | Ownership | Change cost | Proof | Cost to run | |
|---|---|---|---|---|---|
| Prompting + retrieval | Varies by run | Builder owns it | Low per agent, but every agent separately | What the model said | Low |
| Guardrails and filters | High for safety and format; none for business rules | Builder or security | Low | Which filters fired | Low |
| Evals and model-as-judge | Measured, not enforced | Builder | Medium: test sets need updating | Scores, not reasons | Medium to high, grows with loops |
| Observability | None (after the fact) | Builder or platform | Not applicable | What happened | Low to medium |
| Hard-coding or rules engine | High | IT | High: a development cycle per change | Reproducible, if versioned | Low |
| Sealed, executable rulebook | High | Domain owner | Low: once, for every agent | Rule and version per case | Low per decision; discovery up front |
The ratings are our judgment for typical setups, and a careful team can push any row further. Two patterns stand out. The first four rows are cheap and fast, and they all leave the business rule inside the model's interpretation. The two at the bottom take the rule out of the model, and they differ on who owns it and what a change costs.
How to choose
Most teams use several, so the real decision is which approach governs which decisions. Two questions do most of the sorting.
How expensive is a wrong decision? If it is cheap and easy to reverse, prompting, guardrails and monitoring are enough, and adding more costs more than it saves. If it moves money, changes a record or commits the company to a customer, the rule needs to be checked before the action, outside the model.
How often does the rule change, and who changes it? If the rule is stable and IT is happy to own it, hard-coding is fine. If it changes every quarter, lives in policies the business owns, or differs by product, region or client, a developer-owned rule will fall behind, and the owner needs to be able to change it directly.
Put together:
- Low stakes, any rate of change: prompting and retrieval, guardrails, monitoring.
- High stakes, rules that rarely change, few of them: hard-code them, version the code, log the inputs.
- High stakes, rules that change or conflict, owned by the business: a sealed rulebook for the decisions, with guardrails, evaluations and monitoring kept around it.
The approaches also stack. A well-built agent in regulated work usually has guardrails on its inputs and outputs, an evaluation set for its language tasks, monitoring on everything, and a rulebook check in front of the handful of decisions that can't be taken back.
The strongest objection: "We'll build it ourselves"
An AI lead at a bank put it directly: a two-person team writing ordinary code can move faster than any platform. For the first agent, that may well be true, and we'd rather say so than pretend otherwise.
The harder question is the second year. The first agent's rules fit in one developer's head. By the tenth agent, the same refund limit or authority level is written in several codebases, and when the policy changes, someone has to find every copy. Each conflict between documents gets settled by whichever developer hit it first, in code the business owner never reads. And when an auditor asks which version of the rule governed a decision eight months ago, the answer depends on how carefully each team logged. The first agent is easy. The hundredth rulebook version is the work.
If you build it yourselves, build those parts deliberately: one place for the rules, an owner who approves each change, a version on every decision, and a way to replay old cases against a new rule before it goes live.
What to do on Monday
- List the approaches your agents use today. Most teams will find three or four. Write down which decisions each one actually governs.
- Find the decisions that sit only behind prompts and monitoring. For each, ask what a wrong one costs and how you would know. Our post on the cost of agent errors has a worksheet for this.
- For those decisions, find the rule and its owner. If the rule exists in more than one place, or nobody can name the owner, that is the gap, whichever approach you pick.
- Decide who should be able to change the rule. If the answer is "the business", a developer-owned implementation will fall behind. If the answer is "IT, and it changes twice a year", hard-coding is fine.
- Keep the rest. Whatever governs the decisions, keep the guardrails, evaluations and monitoring around them.
Where Factify fits
Factify is the sixth approach, so here is how it works in our own terms.
Rulebook Studio pulls the rules out of your policy, procedures, regulation and tribal knowledge, and flags every conflict and contradiction. The domain owner reconciles them and seals one versioned truth.
The Sealed Truth becomes an Executable Factified Rulebook that agents use as tools, through MCP or API, on any platform and with any model. The Rulebook directs each step and checks it. Nothing leaves a step unless it matches, and judges step in at owner-defined points or when a case isn't covered.
Every case is recorded in the Ledger with the Rulebook version it ran on.
It sits alongside the other five, not instead of them. Keep your guardrails, your evaluations and your dashboards. Factify takes the business rule out of the model for the decisions where that matters.
Not sure which approach fits your process? Bring one process and we'll sort its decisions with you.
Book a sessionFrequently asked questions
What are AI guardrails?
AI guardrails are checks that run around a model to block unsafe, off-topic or badly formatted inputs and outputs, such as personal data, prompt injection or toxic content. They are essential for safety and format, but they are not designed to enforce business rules like approval limits or authority levels.
What is the difference between guardrails and a rulebook for AI agents?
Guardrails check what an agent says. A rulebook defines what an agent is allowed to do in a business process, step by step, and checks each step before it moves on. Most production agents need both.
Is LLM-as-a-judge enough to keep an agent on policy?
It improves quality and catches some errors, but a judge model is a second probabilistic reading of the same rule, so it can't guarantee the same decision every time. It also adds inference cost with every loop. It works best as a quality check, not as the control for high-stakes decisions.
When is hard-coding business rules the right choice?
When there are few rules, they rarely change, and IT is the right owner. Hard-coded rules are consistent and reproducible. They become expensive when rules change often, are owned by the business, or conflict across documents.
Do I need a sealed rulebook for every AI agent?
No. Agents doing low-stakes, reversible work are well served by prompts, guardrails and monitoring. A sealed rulebook earns its place on decisions that move money, change records or commit the company, where a wrong answer is expensive and has to be provable.