OpenTelemetry for Portable AI Agent Evaluation Traces

Almost every AI evaluation tool on the market starts with the same onboarding step: “Install our Python SDK.”

You import their custom wrapper, decorate your functions, and pipe your agent's core execution through their proprietary client. It works fine on day one. But by day thirty, your codebase is tangled up in vendor-specific tracing logic. If you want to try another tool or stream traces to your production observability backend, you're looking at an invasive refactor.

For multi-turn agents, that vendor lock-in is a dead end.

The industry didn't need another proprietary tracing format—it needed a standard. And just like it did for microservices and cloud infrastructure a decade ago, the ecosystem has converged on OpenTelemetry.

You can't evaluate an agent from a final response

Evaluating an LLM completion used to be simple: compare an input prompt to an output string.

Multi-turn agents don't work that way. An agent reasons, queries tools, handles API errors, asks follow-up questions, and changes its plan over multiple turns.

A customer service agent might return a polite, perfectly formatted final answer: “Your refund has been issued.” But looking only at that final message hides critical failure modes:

• Did the agent call issue_refund before verifying the customer's PIN?
• Did it hallucinate an argument when talking to the database?
• Did an intermediate reasoning step leak private customer data into a public prompt?

To evaluate behavior, you need the complete trace: the full sequence of user messages, assistant replies, reasoning scratchpads, tool requests, payloads, and retries.

OpenTelemetry's GenAI semantic conventions standardize this entire execution graph. Instead of every vendor inventing their own JSON schema for tool calls and conversation turns, the industry now shares a unified attribute model.

The ecosystem is already built

The best part of an open standard is that you rarely have to write instrumentation from scratch.

If you're building with popular frameworks like LiteLLM, LangChain, LlamaIndex, or CrewAI, they already emit standard OpenTelemetry spans. If you want drop-in auto-instrumentation across multiple LLM providers, open-source projects like OpenLLMetry (from Traceloop) and OpenInference handle it with a couple lines of config. And if you have custom internal routing, official OpenTelemetry SDKs exist for every major language.

Because Halios natively ingests standard OTLP (OpenTelemetry Protocol) traces, any of these tools work immediately. You can stream traces directly to Halios or route them through an existing OpenTelemetry Collector.

Stop adding SDK baggage to your codebase

Insisting on a custom SDK to run evaluations is unnecessary friction. It adds dependencies you don't need and locks your core application logic to an external vendor.

Using OpenTelemetry flips the dynamic:

1. If you already have telemetry: You don't touch your application code. Point an OTLP exporter to Halios, and you can start running evaluations on your existing traces.
2. If you're instrumenting for the first time: You're making a portable investment. The traces you create today can power your behavioral evals in Halios and feed your operational dashboards in Datadog, Grafana, or Honeycomb.

This is why the Halios toolkit is designed to be lightweight. We don't force you into a proprietary runtime or require you to adopt a custom framework. You use whatever tools fit your stack, and you keep your telemetry open and portable.

One trace, two jobs

Observability and evaluation aren't competing paradigms—they are two ways of looking at the same trace.

Your observability stack uses traces to answer operational questions: What is our p99 latency? Why did that database query time out? How many tokens did this workflow burn?

Halios uses those same traces to answer behavioral questions: Did the agent follow policy? Did it verify identity before calling sensitive tools? Did a prompt change cause subtle regressions across twenty scenario trials?

When you base your agent architecture on OpenTelemetry, you get both answers from a single, portable trace.