Most teams do not avoid AI agent evaluations because they doubt their value. They avoid them because building and maintaining a useful evaluation suite competes with shipping the agent itself.
The usual workflow makes that problem worse. A developer leaves the repository, opens a separate evaluation product, reconstructs how the agent works, copies examples into forms, and manually keeps those definitions aligned with changing code.
Halios makes a different choice: the coding agent is the primary interface for building and maintaining evals.
That choice matters because the coding agent already has the context an evaluation workflow needs. It can inspect the agent entrypoint, tool schemas, system prompts, policies, error handling, and existing tests in the repository. Halios contributes the evaluation model: scenarios, checks, fresh trials, trace evidence, and repeatable evaluation runs.
The minimal workflow
The starting prompt can be short:
Run npx skills add HaliosAI/halios --skill halios, then use the Halios skill to set up evals for this agent.The prompt describes the outcome. The Halios skill provides the procedure.
Working from the repository, the coding agent can:
- Identify the agent runtime and the behaviors that matter.
- Find tool contracts, policies, and likely failure boundaries.
- Draft realistic scenarios and evaluation checks.
- Configure trace export using standard OpenTelemetry instrumentation.
- Run a baseline and inspect the resulting failures.
- Leave the evaluation definitions in the repository for review.
This is more than generating test text. The useful part is connecting evaluation design to the implementation that will actually be exercised.
Repository context produces better evals
Consider a refund agent with tools for looking up an order, verifying identity, checking eligibility, and issuing a refund. A generic eval generator might create examples about polite responses or correct refund policy language.
A coding agent can see the operational sequence in the code. It can discover that identity verification must happen before a refund tool call, that refunds must use the original payment method, and that the tool expects an order_id rather than a free-form customer description.
Those details lead to stronger evaluation criteria:
- Does the agent verify identity before taking an irreversible action?
- Does it preserve the required tool ordering under user pressure?
- Does it pass arguments that match the actual tool schema?
- Does it refuse a policy exception while still helping the user?
The resulting suite evaluates the agent you built, not an abstract chatbot.
What Halios does—and what the coding agent does
The boundary is deliberate.
The coding agent has repository access and can propose changes to prompts, tools, policies, and tests. Halios does not need your source code. Halios receives the agent traces required to run checks, group trials, calculate results, and provide evidence about what happened.
| Responsibility | Coding agent | Halios |
|---|---|---|
| Understand repository structure | Yes | No source access required |
| Author scenario and check definitions | Yes | Validate and execute them |
| Run the agent | Through the project harness | Coordinate and record trials |
| Inspect behavior | Read trace evidence | Store traces and check results |
| Propose a fix | Yes | Measure whether it improved behavior |
Evals become part of normal development
Once the suite exists, maintenance becomes conversational. A developer can ask the coding agent to add an adversarial scenario for a newly discovered risk, explain why a check failed, rerun affected trials, or prepare a CI quality gate.
The interface remains the workflow developers already use. The evaluation definitions remain visible in Git. The evidence remains available in Halios.
That is what coding-agent-native means in practice: not a chatbot added to an eval dashboard, but an evaluation system designed around the coding agent as the place where development work begins.