Designing LLM Systems for Production: Evals, Guardrails, and the Failure Modes Nobody Demos

The demo is usually the easy part.

You wire up a prompt. You pass in some context. The model gives you something impressive. It summarises a document, drafts a reply, classifies a support ticket, generates some structured output, or takes a messy instruction and turns it into something useful.

That first moment is genuinely exciting. I still enjoy it.

But it is not production.

Production starts when real users arrive with messy inputs, partial context, unclear intent, adversarial behaviour, latency expectations, cost constraints, privacy requirements, and a very different standard for being wrong.

That is where a lot of LLM systems struggle.

Not because the model is useless. The model can be very useful. The problem is that too many systems are designed as if the model is the system.

It is not.

The model is an unreliable dependency inside the system. A powerful one, but still a dependency. It can be slow. It can be expensive. It can hallucinate. It can drift. It can ignore instructions. It can return something that looks correct and is not.

The engineering work is not finding the perfect prompt. The engineering work is designing the system around that reality.

Non-determinism is a design constraint

The first mental shift is accepting that non-determinism is not a bug you are going to fully eliminate.

The same input can produce different outputs. A small prompt change can move behaviour in ways you did not expect. A model upgrade can improve one class of answer and quietly degrade another. A retrieval result can change the shape of the response. A user can phrase the same intent in ten different ways.

Traditional software has uncertainty too, but LLMs bring it closer to the centre of the product.

That changes how you design.

You cannot rely on a single happy-path example. You cannot ship because it worked well in a notebook. You cannot treat a prompt as a fixed piece of logic and assume it will behave like a function.

The system has to assume variation.

That means validating outputs, not trusting them. It means designing fallbacks. It means measuring behaviour over a set of examples rather than arguing from one impressive response. It means being very clear about where the model is allowed to be creative and where it must be constrained.

Some parts of a product can tolerate fuzziness. Others cannot.

A brainstorming assistant can be a little loose. A customer-facing action, a compliance-sensitive summary, a financial instruction, a permission change, or anything that touches real-world state needs a different standard.

The architecture should reflect that.

Evals are the test suite

If prompts and models are part of the behaviour of the system, then evals are the test suite.

I do not mean a vague sense that "we tried it and it looked good." I mean a curated set of cases that represent the actual work the system needs to do, including the awkward cases.

The common inputs. The edge cases. The ambiguous requests. The hostile requests. The boring cases that should not regress. The examples where a hallucination would be expensive. The cases where the right answer is to refuse, ask for clarification, or do nothing.

That set becomes one of the most important assets in the system.

Every prompt change should run against it. Every model change should run against it. Every retrieval change should run against it. Every change to the surrounding workflow should be checked against the behaviours you care about.

The point is not to prove the system is perfect. It will not be.

The point is to stop flying blind.

This is also where teams need to be honest about LLM-as-judge — using a model to grade another model's output. It can be useful. It can help score tone, completeness, format, and semantic similarity at scale. But it inherits some of the same failure modes as the thing it is judging. It can be inconsistent. It can miss subtle correctness issues. It can reward plausible answers.

So I treat it as a useful signal, not the source of truth.

For high-risk behaviours, I still want deterministic checks where possible: schema validation, exact constraints, policy rules, reference checks, human review, and targeted assertions around the things that matter.

The eval system should not just say "good" or "bad". It should tell you what changed.

Did the model become more verbose? Did it stop following a formatting rule? Did it start inventing missing fields? Did it become more willing to take unsafe actions? Did it miss an edge case it used to handle?

That is how you catch silent regressions before users do.

Guardrails are not decoration

Guardrails are often treated like a layer you add after the product works.

I think that is backwards.

Guardrails are part of the product design.

They start at the input. What are users allowed to ask the system to do? What context does the model receive? What sensitive data should never enter the prompt? What instructions from retrieved content should be treated as data rather than authority? What does the system do when the user's intent is unclear?

Prompt injection and jailbreaks are not abstract security theatre. If your system retrieves untrusted content, accepts user-provided text, or connects model output to tools, you have to assume someone will try to manipulate the model.

That does not mean you can make the model perfectly obedient. You cannot.

It means you reduce the blast radius.

You separate instructions from data. You minimise sensitive context. You constrain tool access. You validate arguments before execution. You require approval for high-stakes actions. You keep permissions outside the model. You make the model propose; the system decides what is allowed.

Output guardrails matter just as much.

Where possible, force structured output. Validate it. Reject invalid states. Do not let a paragraph of confident prose become an unchecked command. If the model is producing structured data, parse it strictly. If it is selecting an action, verify that action exists and the user is allowed to perform it. If it is summarising source material, make it stay grounded in the supplied context.

A model can be persuasive. That is useful in a writing tool. It is dangerous in a control plane.

The failure modes nobody demos

Most demos show the happy path.

Nobody demos the model hallucinating a policy and saying it with confidence. Nobody demos the 99th percentile latency spike. Nobody demos the provider outage. Nobody demos the monthly bill after usage grows. Nobody demos the model version changing underneath you and degrading one important workflow while the rest of the product looks fine.

But those are the things production systems need to absorb.

Confident hallucination is the obvious one. The answer looks right, reads well, and is wrong. Grounding helps here. Retrieval helps. Source constraints help. But none of them remove the need to verify where the answer came from and what it is allowed to claim.

Latency is another. LLMs are often slow compared with normal application paths. If the interaction blocks the user, you need a budget. If the work can happen asynchronously, design it that way. If the model call is optional, have a degraded path. Timeouts are not an afterthought.

Cost can get away from you quickly. Long context windows, repeated retries, chatty agent loops, oversized prompts, and inefficient retrieval can turn a promising feature into an expensive one. Cost needs to be designed like any other production constraint: budgeted, measured, and visible.

Provider outages are also real. Even if you trust the provider, your system should know what happens when the dependency is slow, unavailable, or returning errors. Retry carefully. Fall back where you can. Fail closed where safety matters. Do not let an external model outage become an uncontrolled product failure.

And then there is silent degradation.

This is one of the risks I think teams underestimate. A model can change, a prompt can be adjusted, a retrieval index can shift, or a dependency can be upgraded, and the system still "works" in the shallow sense. It returns responses. It passes basic smoke tests. Nothing crashes.

But the quality has moved.

That is why evals and observability matter. Without them, you may not know the system got worse until users lose trust.

Approval-first is a safety pattern

One pattern I like, especially for action-oriented AI, is approval-first.

The system captures intent. It prepares the action. It shows the user exactly what will happen. Then it executes only after acceptance.

That is the model I care about for products like Keldr: do not hide the action behind the magic. Capture the user's intent, structure it, make the next step clear, and keep the user in control before anything meaningful happens.

This matters because a lot of AI product design optimises for removing friction.

Sometimes that is right. But in high-stakes workflows, a little deliberate friction is not a failure. It is the control surface.

Sending an email, changing a record, scheduling something, updating a customer account, triggering a workflow, modifying permissions — these are not the same as generating a draft.

The system should make the proposed action visible. It should show the recipient, the content, the timing, the destination, the affected object, or whatever else matters. It should give the user a chance to correct it. And only then should it execute.

That pattern turns the model from an autonomous actor into a controlled assistant.

I think that distinction matters.

Observability is how you debug behaviour

You cannot debug what you did not capture.

For LLM systems, observability has to include more than normal application logs. You need to know what prompt was sent, what context was included, what model was used, what output came back, what validation happened, what tool calls were proposed, what was accepted or rejected, how long it took, and what it cost.

That does not mean logging sensitive data carelessly. Privacy and retention rules still matter. In some systems, you will need redaction, sampling, access controls, or separate handling for sensitive prompts and outputs.

But the principle stands: if the system behaves badly, you need enough traceability to understand why.

Was the retrieved context wrong? Was the prompt ambiguous? Did the model ignore a constraint? Did validation fail and retry into a worse answer? Did latency come from retrieval, the model, or downstream tooling? Did a cost spike come from longer inputs, more retries, or a new workflow?

Without traces, teams end up debugging vibes.

That is not good enough for production.

The system around the model

The most reliable LLM systems I have seen do not pretend the model is reliable.

They contain it.

They use retrieval to ground answers where facts matter. They use structured outputs where the system needs to act. They validate. They budget. They time out. They fall back. They evaluate changes before shipping them. They log enough to understand behaviour. They keep humans in the loop where the risk justifies it.

That does not make the product boring. It makes it usable.

The magic is still there. But it is wrapped in engineering.

That is the difference between a demo and a system I would put my name on.

A demo asks, "Can the model do this?"

Production asks, "What happens when it does the wrong thing, slowly, expensively, confidently, or differently tomorrow?"

That is the question I want teams asking earlier.

Treat the model as an unreliable dependency you must contain. The engineering is in the system around it — evals, guardrails, budgets, fallbacks, observability, and approval where it matters.

Not in the perfect prompt.