LLM evals developer tools
Every listing tagged evals, A to Z: 9 tools across 1 category. Listed in alphabetical order, not ranked.
Arize Phoenix
Arize's AI observability platform, source-available under the Elastic License 2.0: OpenTelemetry tracing through OpenInference for OpenAI, Anthropic, LangGraph, the Vercel AI SDK and more, with LLM evals, datasets, experiments and a prompt playground; runs locally, in Docker or on Kubernetes.
Braintrust
Observability and evals platform for AI agents: trace production calls from OpenAI, Anthropic, Gemini, the Vercel AI SDK, LangChain and more, run experiments scored by LLM judges, Autoevals or your own code, review in playgrounds and human annotation queues, and let the built-in Loop agent generate test cases and build evals. MCP server and a bt CLI. Free Starter plan with unlimited users; Pro $249 a month.
Laminar
Open-source (Apache-2.0) observability for AI agents, written in Rust: one-line OpenTelemetry tracing for the Vercel AI SDK, Browser Use, Stagehand, LangChain, OpenAI, Anthropic and more, Signals that flag behaviour you describe in plain English, evals from an SDK and CLI, browser session recordings, SQL over traces and MCP access for coding agents. Self-host with Docker Compose, or use the cloud, free for 1 GB of data.
Langfuse
Open-source LLM engineering platform: traces and sessions for LLM apps and agents with token and cost tracking, prompt management, datasets, LLM-as-judge and human evals; OpenTelemetry-based, self-host free or use Langfuse Cloud in the US, EU or Japan.
LangSmith
LangChain's platform for agents: tracing and online and offline evals for any framework, plus managed agent deployment, sandboxes and an LLM gateway; cloud in the US or EU, hybrid or self-hosted on Enterprise.
Latitude
Open-source (MIT) observability and quality platform for AI agents: traces production runs, groups failures into tracked issues with evals attached, dispatches your coding agent to fix them and replays the fix against the failing traces. MCP server and CLI. Free Starter plan with 20K credits a month; Pro $99 a month.
Opik
Comet's open-source (Apache-2.0) LLM observability and evaluation platform: traces for agents and multi-turn conversations, datasets and experiments with 30+ LLM-as-a-judge and heuristic metrics, pass/fail test suites, a PyTest integration for CI, online evaluation rules on production traces, guardrails and an agent optimizer. The full platform, backend included, self-hosts free; Opik Cloud is free for 25K spans a month.
Promptic
Optimization platform for GenAI apps: OpenTelemetry-native LLM tracing, model benchmarking and one-click prompt and agent tuning scored on your own quality and cost metrics; free tier, team plans from €149/month.
W&B Weave
Weights & Biases' toolkit for tracing and evaluating AI agents: sessions, turns, tool calls and sub-agents as first-class traces, an evaluation framework with comparisons and leaderboards, built-in signals with Slack alerts, guardrail scorers for toxicity, PII and hallucinations, a playground, and an MCP server so coding agents can read traces and run evals. SDK is open source (Apache-2.0); free plan with 1 GB of trace data a month, Pro from $60 a month.