When an agent's evaluations live only in a web dashboard, they quickly drift away from the codebase. Prompts, tools, and application logic evolve in Git, but the rules defining how the agent should behave remain trapped in an external UI.
Halios keeps evaluation scenarios and checks directly in your repository. That brings evaluations into the same day-to-day workflow you already use to build, branch, and review code.
Keep the specification close to the agent
When test scenarios live in the repo, your coding agent gets full context on how the system should actually behave.
When you ask it to tweak a prompt, add a tool, or fix a bug, it can read existing scenarios, propose new edge cases, and run tests locally. You don't have to copy-paste context back and forth between your editor and a dashboard.
Human developers get the same benefit. Evals branch when your code branches, travel in the same pull request, and get reviewed by the team before merging.
Review behavior and code together
Say you're adding a cancel_subscription tool.
In a single pull request, you can include:
- The tool implementation.
- Test scenarios covering standard user requests and edge cases (like unverified users).
- The checks that must pass (like verifying identity before canceling).
- Fresh evaluation results from the branch.
Reviewers can verify both the feature and its test criteria in one diff. If a policy changes later, git log shows exactly when and why.
Why not write every check as custom code?
If keeping evals in Git makes sense, why not write every test as custom Python or TypeScript?
If you have unique business logic, custom scripts make sense. But for everything else, writing eval harnesses from scratch is a maintenance trap.
Writing a check is the easy part. The real work is running multi-turn simulations, handling model rate limits, capturing traces, and tracking regressions over time. If you hand-roll all of that in your repo, your team ends up maintaining an eval infrastructure instead of building your product.
Specification in Git. Execution in Halios.
Halios separates specification from execution:
Your repo holds a clean, versioned description of the scenarios and rules that matter. Halios handles the execution—simulating multi-turn conversations, running judges, collecting traces, and tracking pass rates.
Your coding agent gets the context it needs in the editor. Your team gets standard Git reviews and PR workflows. And Halios does the heavy lifting under the hood.
Your repository defines what good behavior looks like. Halios runs the pipeline that tests it.