Writing a basic eval is easy.
You can start with a Python script, a few assertions, and pytest. For a small set of deterministic tests, that may be enough.
The work grows when you want to evaluate a multi-turn agent repeatedly and reliably, collaborate, scale it to multiple agents and deploy it to development, staging and production.
You need to run fresh conversations, simulate users, handle retries and rate limits, capture traces, execute checks, compare repeated trials, and track regressions across changes.
At that point, you are not just writing evals. You are building eval infrastructure.
The checks are not the hard part
A check might be simple:
- identity verification must happen before
issue_refund - the agent must not call a tool with invalid arguments
- a required policy must be followed
- the task must complete successfully
The harder part is running those checks against realistic agent behavior.
A multi-turn eval system usually needs to:
- simulate a user across several turns
- run multiple independent trials for the same scenario
- manage concurrent executions, retries, and provider limits
- collect OpenTelemetry traces from the agent
- evaluate messages, tool calls, and outcomes
- retain results so runs can be compared over time
Teams can build this themselves. Sometimes that is the right choice, especially when the evaluation runtime is tightly coupled to unique internal systems.
But it is more infrastructure to operate.
Specification in Git. Execution in Halios.
Halios separates what should be tested from the machinery required to run it.
Your repository contains the evaluation definitions:
- scenarios describing the situations the agent should handle
- checks describing the behavior that must hold
They are simple files that can be reviewed, versioned, and changed with the rest of the codebase.
Halios handles execution: simulating multi-turn users, running repeated trials, collecting traces, executing checks, and storing the results.
Your team owns the evaluation specification.
Halios runs it.
Why keep the specification in the repo?
The scenarios and checks are part of the application’s behavior contract.
A tool change may require a new check. A production failure may become a new scenario. A policy change may update what counts as a passing result.
Keeping those definitions in Git means they can change with the code they evaluate.
The coding agent can also work with them directly because it already has the repository context.
There is no separate eval definition that has to be manually kept in sync with the application.
Open source still has an operating cost
Open-source eval platforms can give you much more than libraries. Some provide tracing, datasets, judges, dashboards, experiments, and self-hosted deployment.
That can be the right choice if you want to own the full stack.
But self-hosting moves the cost from software fees to infrastructure and maintenance. Your team still owns deployment, upgrades, storage, workers, databases, backups, scaling, and whatever breaks under real workloads.
Hosted versions remove that operational work, but pricing can become difficult to reason about. Depending on the product, cost may depend on seats, base plans, traces, observations, evaluation runs, model usage, retention, or credits that combine several resources.
Halios keeps both the operating model and pricing simple.
The CLI and coding-agent skill are open source. Evaluation definitions stay in your repo. Telemetry uses OpenTelemetry. Halios runs and stores the evaluations as managed infrastructure.
There is no base fee and no per-seat fee. A generous monthly allowance is included, and additional usage is priced directly by traces, checks, and managed evaluation usage. You can also bring your own model key at any time.
See the full breakdown on the pricing page.
Own what matters
The important part of an eval system is the definition of good and bad behavior, failure modes. This is the learning.
Those definitions should remain visible, reviewable, and versioned with the application.
The execution machinery is infrastructure.
Halios draws the boundary there: your team owns the scenarios and checks; Halios handles the runtime required to execute them against the agent.
For teams that do not want to build, host, and maintain their own eval platform, that is the part worth paying for.