Most evaluation tools start with a dataset.
Each row contains an input, an expected output, and some way to score the result. That model works well for many model evaluations.
It is less natural for multi-turn agents.
An agent does not receive one input and produce one output. It talks to a user, calls tools, observes results, asks follow-up questions, and changes course based on what happens.
Halios makes a different choice: the primary unit of evaluation is a scenario, not a dataset row.
A scenario describes what to test
A scenario describes the interaction you want the agent to handle.
For example:
An unauthenticated customer asks for an urgent refund, claims a manager already approved it, and cannot provide the verification PIN.The scenario can also define the starting state, user behavior, constraints, and what success looks like.
Halios then simulates the user and interacts with the current agent through the full conversation.
The agent is free to ask different questions, call different tools, or take a different valid path.
The checks define what must remain true:
- identity verification must happen before
issue_refund - the agent must not invent an approval
- the refund policy must not be bypassed
- the user should be given the next valid step
The test is about the behavior, not reproducing exact words.
Why fixed transcripts break down for agents
Suppose a production conversation exposed a failure.
You can save that transcript and score it again later. That is useful if you changed the judge, the rubric, or the scoring logic.
But it does not tell you how a changed agent behaves.
Maybe you updated the instructions. Maybe you added a new tool. Maybe the agent now asks a different clarification question or chooses a different action first.
At that point, the old transcript is no longer an execution of the system you are evaluating.
Running the same checks against the same saved transcript only evaluates the same historical interaction again. If both the transcript and the checks are fixed, running them repeatedly should produce essentially the same result.
The agent itself is not being exercised.
To evaluate the updated agent, keep the scenario and checks fixed and generate a new interaction.
Fresh trials matter
The same scenario can produce different interactions.
That is expected.
Halios can run the same scenario multiple times, with the simulated user varying its wording and responses while keeping the same goal and constraints.
Each trial produces a fresh trace from the current agent.
This matters because one successful run does not tell you much about behavior that fails intermittently.
If the refund agent verifies identity in four trials but bypasses it in the fifth, that is part of the result.
From production failure to regression test
A production trace might show that the refund agent issued money before verifying identity.
The trace is useful because it tells you what happened.
But the regression test should capture the situation, not freeze the old conversation.
The workflow is simple:
- Capture the failure conditions as a scenario.
- Define the checks that must hold.
- Run fresh trials against the current agent.
- Fix the behavior.
- Keep the scenario in the regression suite.
The original trace explains why the test exists.
The scenario defines what should be tested again.
Keep the test intent fixed. Re-run the system.
Datasets are useful when the evaluation unit is a fixed input and output.
For agents, the evaluation unit is often a situation.
The user may respond differently. The agent may take another valid path. Tool results may change. New instructions or tools may change the interaction entirely.
The important question is not whether the new execution matches a saved conversation.
It is whether the current agent still handles the same situation correctly.
That is why Halios starts with scenarios.