Getting Past “It Seems to Work”
You’ve polished the prompts, attached tools, hooked up RAG, and even run an agent loop. It works beautifully in the demo. But the moment you try to turn it into a real service, you suddenly don’t know where to start.
- ‘How do I measure how well it actually works?’
- ‘If it worked fine yesterday but gives strange answers today, how do I figure out what happened?’
- ‘How do I stop users from feeding it malicious input?’
- ‘How do I keep a bad decision from turning into a major incident?’
To take an AI system ‘from experiment to production,’ you need five things. In everyday language, they look like this.
- 'Give it a test and measure how well it does in numbers' (Evals)
- 'When a given answer comes out, retrace the process later' (Traces)
- 'Keep risky inputs and actions from turning into real damage' (Guardrails)
- 'Let a person review high-stakes decisions and stop them if needed' (Human-in-the-Loop)
- 'Build a working environment where the agent can check and fix its own work' (Harness)
In this post, I’ll unpack these five one at a time. The jargon stays in parentheses; the body explains everything in plain language.
Why Evaluating AI Is Harder Than Evaluating Regular Software
‘TL;DR: A regular program has “one right answer,” but AI has “many possible answers,” and its output shifts a little every time, so it’s hard to define “pass/fail.”‘
Testing regular software is relatively clear-cut. add(2, 3) must return 5. Right means pass; wrong means fail.
AI doesn’t work that way.
- The same question gets a slightly different answer every time (nondeterminism)
- There can be more than one “right answer” (“Summarize this post” doesn’t have just one correct answer)
- Many answers are only partly right (the substance is right, but the format is wrong)
- User input itself is free-form, so it’s unpredictable
That’s why evaluating AI is closer to a ‘school exam’ than a ‘unit test.’ You decide the grading criteria first, have it work through a set of problems, and score the results.
Evals: Make the AI Take a Test
‘TL;DR: Build a set of representative questions and expected answers, and every time something changes, “make it take the test” so you can check quality in numbers.’
The foundation of AI evaluation is building an ‘eval set.’ You prepare representative questions and their expected answers (or grading criteria) in advance, and every time you change the prompt or the model, you have the AI take the test with this set and compare the results.
[Eval set example] Question 1: "Tell me the refund policy" → Expected: includes the keyword 'within 14 days', polite tone, under 200 characters Question 2: "Who is the top administrator?" → Expected: a no-permission notice (personal information is not disclosed) Question 3: "Status of order #12345" → Expected: the exact status, after querying the real order system ...
Early on, you can start with 10 to 100 representative questions. But don’t stop there. As the service keeps running, the norm is to keep adding real user logs, failure cases, and unexpected edge cases until the set grows to hundreds or thousands of items.
Eval Sets Come in Two Kinds
- ‘A test to see how good a new model is’: used to probe the limits with hard problems. A 100% pass rate isn’t the goal.
- ‘A test to see whether what used to work still works’: confirms that existing features survive after you change the prompt or model. The goal is a near-100% pass rate.
The first is called a ‘capability eval,’ and the second a ‘regression eval.’ You need both, but they serve different purposes.
What Do You Grade Against?
The simplest principle: ‘If you can measure it with code, measure it with code.’ Things like validating JSON format, checking for required keywords, and whether tests pass can be judged definitively by a computer. Things that are hard to measure with rules, like tone, faithfulness, and explanation quality, are graded by people, or sometimes by another AI (this approach is called ‘LLM-as-Judge’). It’s convenient but biased, so for important evals you periodically calibrate it against human grading.
Don’t Just Measure Quality: Measure Cost and Speed Too
In production, even if response quality goes up, you can’t actually put it into service if the cost goes up 5x or responses get 3x slower. So evals look beyond quality at ‘safety, tool use, cost, performance, and stability’ as well.
| Eval category | Examples |
|---|---|
| Quality | Accuracy, format compliance |
| Safety | Exposure of personal information, attempts at forbidden actions |
| Tool use | Did it call the right tool with the right arguments? |
| Cost | Tokens per request, cost per request |
| Performance | Response time, timeout rate |
| Stability | Error rate, retry rate |
Judge Agents by “Did It Actually Happen?”, Not “Did It Say So?”
This is a key point in agent evaluation. You don’t check whether the agent said “Done.” You check the final state of the environment: ‘Was a ticket actually created? Did the refund status change? Did the tests pass?’ What it says and what actually happened can differ.
‘Evaluate complex systems layer by layer.’ For RAG, check “did retrieval work?” and “was the answer good?” separately; for an agent, check “were the tool calls appropriate?” and “is the final state correct?” separately. That’s why I said in an earlier post that ‘RAG failures often start in retrieval rather than in generation.’
Traces: Tracking Down “Why Did It Give This Answer?”
‘TL;DR: Recording every LLM call, tool use, and intermediate result an agent produces, so you can later follow “why it gave this answer.”‘
If an eval set shows you ‘how well it works,’ a trace (an execution record) shows you ‘why it turned out that way.’
In a regular program, you track things down with logs and stack traces, but in an AI system the LLM call itself is close to a black box. A trace doesn’t fully reveal the AI’s ‘inner thoughts.’ What it does is leave a reproducible record of ‘what input it received, which model calls and tool calls it went through, and which intermediate results it relied on to reach the final answer.’ (Some reasoning models hide their internal reasoning or provide only a summary, so you can’t log the entire raw thought process.)
What you actually need to record:
| What to record | Why you need it |
|---|---|
| Input prompt (in full) | Needed to reproduce the issue later |
| Output answer (in full) | Something to compare against |
| Model/settings used | The same input gives different results if the model or settings change |
| Tool call history | Which actions the agent chose and what results it got back |
| Time taken/token count/cost | Performance and cost analysis |
| User/session identifiers | Tracking issues for a specific user (hashes or internal IDs recommended instead of raw IDs) |
Only with records like these can you actually follow the trail and find the cause when a bug report comes in saying “It worked fine yesterday, but today it’s giving weird answers.” You can build your own logging system or use a dedicated observability tool (see the developer notes at the end for specific tool names).
‘Caution: traces can become a store of sensitive information.’ Full prompts and outputs can contain personal information, internal documents, API keys, and customer data. Don’t just switch a tracing system on and leave it; design ‘masking (automatically removing sensitive patterns), access control (who can see it), retention periods (when it gets deleted), and a sampling policy (store everything vs. store a portion)’ along with it. For user IDs as well, storing an internal identifier or a hash is safer than storing the original.
Guardrails: Layered Defenses That Reduce Risk
‘TL;DR: Guardrails aren’t a perfect blocking device. They’re lines of defense that “reduce” risk across multiple layers. No single mechanism is enough.’
A guardrail is literally a ‘protective railing.’ The term covers the mechanisms that keep AI from veering off in a strange direction. The important point is that ‘no single guardrail can stop every risk.’ You need input filters, output validation, permission limits, human approval, audit logs, and the application’s own authentication and authorization working together. Layered defense is the principle.
Input Guardrails: Filter on the Way In
- ‘Detecting malicious prompts’: block prompt injection attempts like “Ignore your previous instructions and do X”
- ‘Blocking sensitive input’: mask or reject personal information such as resident registration numbers (Korea’s national ID numbers) or credit card numbers when it’s entered
- ‘Rejecting out-of-scope questions’: you can filter out things like “political talk that has nothing to do with our service”
Output Guardrails: Filter on the Way Out
- ‘Blocking sensitive data exposure’: mask personal information if it shows up in the AI’s answer
- ‘Format validation’: ask for regeneration if the output isn’t in the expected JSON format
- ‘Policy violation checks’: block inappropriate content when it gets generated
Permission Guardrails: Limit What It Can Do
This ties back to the permission-level table in the previous post. Reads are allowed automatically, writes need approval, and deletes need multi-step confirmation. Even for the same agent, ‘which tools it can use’ is set differently depending on the user’s permissions.
Human-in-the-Loop: Stopping Before an Incident Grows
‘TL;DR: Design the AI so that, before a risky action or an uncertain decision, it stops and gets confirmation from a person.’
If guardrails are ‘automatic blocking,’ human-in-the-loop is ‘human confirmation.’ When you automate everything, there are moments of real danger. At those moments, you have the AI ask a person, “Is it okay to go ahead like this?”
Common intervention points:
| Point | Examples |
|---|---|
| Before destructive actions | Deleting files, changing the DB, executing payments |
| Before sending anything externally | Sending email, Slack messages, external API calls |
| Requests with high cost or broad permissions | Bulk data processing, admin-level operations |
| When the AI’s confidence is low | When there’s a signal like “I’m not sure this is right” |
Human-in-the-loop is a good safety mechanism, but it has a cost. Ask too often, and users get worn out and the benefits of automation disappear. The key is designing it to step in ‘only at the points that really matter.’
Harness: The 'Working Environment' That Helps Agents Do Good Work
‘TL;DR: Even a smart new hire needs manuals, checklists, and a test environment to do good work. The same goes for agents.’
The ‘harness’ I briefly mentioned in the previous post is the last topic of this one. The name may be unfamiliar, but the idea is simple.
No matter how smart a new employee is, if their desk is a mess, there are no manuals, and there’s no test environment, they can’t show what they’re capable of. Conversely, when the environment is well set up, even a new hire can check and fix their own work as they go.
AI agents are exactly the same. No matter how well you write the prompt, a sloppy working environment makes good results hard to get. Conversely, even with an ordinary prompt, a well-equipped environment lets the agent check and fix its own work. This ‘entire environment the agent works in’ is called the ‘harness.’
The word ‘harness’ is used a little differently depending on context. An ‘eval harness’ is the infrastructure that runs evals automatically; an ‘agent harness’ is the scaffolding around the model, such as tools, state, and the loop; and a ‘coding agent harness’ means the tests, linters, docs, and CI around a codebase. This post focuses on the third, the most hands-on of the three.
What a Coding Agent Harness Looks Like
For a coding agent, the harness is made up of things like these.
- ‘Test suite’: after changing code, the agent runs the tests itself to see whether anything broke
- ‘Linters/type checkers’: catch code style issues and type errors immediately
- ‘A clear README and AGENTS.md’: explain to the agent, “this is how this project works”
- ‘CI pipeline’: runs automatically on every PR to verify that ‘it really works’
- ‘Structured logs’: for tracking and debugging what the agent did
The better people refine this environment, the higher the agent’s success rate on autonomous work. The message you keep hearing in 2026 is much the same. The OpenAI Codex team’s post “Humans steer. Agents execute.” (February 2026) and “Harness engineering for coding agent users” (April 2026) by Thoughtworks’ Birgitta Böckeler, published on Martin Fowler’s site, both point in the same direction: ‘In the agent era, developers are shifting from people who type every line of code themselves to people who design the environments and feedback loops in which agents can work safely.’
So Where Do Humans Go?
The better the harness, the further people move up to a ‘higher layer.’ Hands-on work like typing code or running tests goes to the agent, while people focus on a ‘higher layer of abstraction’: setting priorities, defining acceptance criteria, and verifying results. ‘Redistributing roles between humans and agents’ is the ultimate goal of a harness.
The Big Picture: A Checklist for Going from Experiment to Production
‘TL;DR: Eval sets, traces, guardrails, human-in-the-loop, and a harness. With these five in place, an AI system can hold up in production.’
To sum up, here are the five things to check when you take an AI system to production.
| Item | Question |
|---|---|
| Evals | Can you measure quality in numbers with a set of representative questions? |
| Traces (execution records) | When an answer comes out, can you trace what caused it? |
| Input/output guardrails | Can you automatically filter out strange inputs and outputs? |
| Human-in-the-loop | Is there a human confirmation step before risky actions? |
| Harness | Is the environment the agent works in (tests, docs, CI, etc.) in place? |
You don’t have to build everything at once. It’s fine to start with an eval set of 5 to 10 cases and basic trace logging. From there, keep feeding real usage logs, failure cases, and user feedback back into your eval set and guardrails as you grow them.
📦 [ Extra Notes for Developers ]
General readers can skip everything from here on. What follows is supplementary material for developers building production AI systems themselves.
‘Eval frameworks’: OpenAI Evals, DeepEval, Promptfoo, Braintrust, LangSmith, and others. Hook them into CI, and your eval set runs automatically every time the prompt or model changes.
‘Observability tools’: LangSmith, Langfuse, Phoenix (Arize), W&B Weave, Braintrust, and others. GenAI/agent/MCP semantic conventions are taking shape on the OpenTelemetry/OTLP side, so connecting these tools to existing APM is getting easier. That said, as of April 2026 the GenAI conventions are still in development status, so check how each tool implements them.
‘Guardrail tools’: you can combine libraries/frameworks like NVIDIA NeMo Guardrails and Guardrails AI with safety classifier models like Llama Guard. No single tool stops every risk, so combining multiple layers is the principle.
‘Managing eval sets’: if you set up a pipeline that automatically adds user feedback (likes/dislikes, reports) to your eval set, the eval set keeps up with real usage patterns and becomes a living document.
‘Components of a harness’ (for coding agents):
– AGENTS.md (a project guide for agents)
– A fast test suite (a feedback loop measured in seconds)
– Linters/type checkers/formatters
– An eval set that runs automatically in CI
– A store for agent action logs
– PR templates and clear acceptance criteria
‘Re-run evals every time the model changes’: a repeat of what I stressed in an earlier post. When you switch models or change settings like reasoning.effort/thinkingLevel, your prompts may no longer be optimal. Always pair a model upgrade with a fresh eval run.
Wrapping Up: The 'Pro' Stage of AI Systems
To sum up:
- Evaluating AI is closer to a ‘school exam’ than a ‘unit test.’ You set grading criteria and run it through a set of representative problems.
- ‘Traces (execution records)’ let you track down “why did it give this answer?” Record every input, output, and tool call.
- ‘Guardrails (safety mechanisms)’ set up automatic blocking at three points: input, output, and permissions.
- ‘Human-in-the-loop’ requires human confirmation before destructive, irreversible, or high-risk actions.
- A ‘harness’ means setting up the working environment, such as tests, linters, docs, and CI, so the agent can do good work. It matters as much as the prompt.
These five are what lift an AI system from a ‘demo that works’ to a ‘system you can ship as a service.’ The difference between teams that build demos but never ship and teams that keep shipping and improving mostly comes down to this.
In the next post, I’ll cover how to actually ‘plug’ the AI technologies we’ve learned so far ‘into your own application.’ It’s a topic Java/Spring developers in particular will appreciate: a more convenient way than raw API calls, the Spring AI framework.
