AI agent teams often accumulate three different reliability systems.
Local testing is a collection of prompts run by hand. CI uses a small deterministic test suite because realistic agent tests are difficult to automate. Production has traces and dashboards that reveal failures only after users encounter them.
Each system contains part of the truth, but they do not share a common evaluation model. A behavior tested locally may never become a merge gate. A production failure may remain a dashboard screenshot instead of becoming a regression test.
Halios is built around a simpler principle: the same scenarios and checks should follow an agent from local development to CI and production.
What stays consistent
A Halios evaluation suite defines two durable ideas:
- Scenarios describe the behavior, starting context, user intent, and constraints the agent must handle.
- Checks define what must be true about the resulting trace, messages, tool calls, or outcome.
Those definitions can be stored in .halios/scenarios.yml and .halios/eval.yml alongside the application. Every evaluation run freezes the relevant scenario and check definitions so results can be understood against the exact suite that produced them.
The trigger changes across environments. The evaluation language does not.
| Environment | Trigger | Primary question |
|---|---|---|
| Local | Developer or coding-agent command | Did this change improve the behavior? |
| CI | Pull request or release workflow | Did this change break a required behavior? |
| Production | OpenTelemetry traces | Is the live agent still satisfying the same expectations? |
Local: shorten the feedback loop
Local evaluation should be fast enough to run while a developer is changing a prompt, tool description, or policy.
The coding agent can select the affected scenarios, execute fresh trials against the current branch, and inspect trace evidence when a check fails. The result is more useful than comparing a final answer string: the developer can see the tool call, intermediate decision, or message that caused the failure.
CI: turn critical behavior into a gate
CI should not run every conceivable experiment. It should protect the behaviors that must not regress.
A team can promote high-value scenarios into a pull-request suite and require critical checks to pass before a change is merged. Because the definitions are versioned with the branch, reviewers can examine the code change and the evaluation change together.
The gate can focus on completion of required scenarios, pass rates across repeated trials, and hard checks for policy or tool-use invariants. An incomplete trial should not quietly disappear from the denominator; missing evidence is itself a failed attempt to verify the behavior.
Production: evaluate real trajectories
Production is where unexpected users, tool states, and multi-turn paths appear. Halios accepts OpenTelemetry-compatible traces and applies evaluation checks after traces are finalized.
The purpose is not to replay production conversations as if they were future executions. Production traces provide evidence and provenance. When a trace exposes a new failure mode, the team can turn its starting conditions into a scenario and run a fresh interaction against the current agent.
That closes the gap between monitoring and testing:
- A production trace fails a check.
- The team inspects the trace and identifies the behavioral boundary.
- The coding agent creates or updates a scenario in the repository.
- The scenario becomes part of local and CI evaluation.
- The fix is verified through new trials before release.
One model, different levels of confidence
Local, CI, and production should not be identical in volume or frequency. Local runs may be narrow and fast. CI may use a stable regression subset. Production may evaluate sampled or policy-relevant traces.
They should, however, speak the same language. When scenarios, checks, traces, and outcomes retain the same meaning across the lifecycle, evaluation stops being a pre-launch exercise and becomes part of how the agent is operated.