“Langfuse | Open-source observability and evaluation | Teams wanting self-hosting and flexibility”
“Langfuse | Open-source observability and evaluation | Teams wanting self-hosting and flexibility”
“One particularly useful capability: you can turn real production traces into evaluation datasets and rerun them against new agent versions.”
“Langfuse also supports running experiments in CI and failing a build when evaluation scores fall below predefined thresholds. langfuse.com”
“Promptfoo — my strongest recommendation here”
“Promptfoo is an open-source evaluation and red-teaming framework.”
“Particularly useful: you can integrate its tests into CI/CD so that security regressions are detected before deployment. promptfoo.dev”
“It also supports trace-based testing, which means you can inspect what tools the agent actually called rather than merely judging its final response. promptfoo.dev”
“Braintrust | Evaluation experiments and regression detection | Teams with evaluation-driven development”
“Braintrust — best for systematic regression testing”
“Its evaluation-first approach makes it attractive for teams that want measurable release criteria rather than relying on manual inspection. Braintrust”
“I'd strongly recommend instrumenting your agents using OpenTelemetry conventions wherever possible.”
“This gives you a more portable instrumentation layer, reducing dependency on any one observability vendor. MLflow AI Platform”
“Weights & Biases Weave | Experiment tracking and trace evaluation | Teams already using W&B”
“Weights & Biases Weave | Experiment tracking and trace evaluation | Teams already using W&B”
“Comet Opik | Agent tracing and evaluation | Teams wanting an open-source option”
“Comet Opik | Agent tracing and evaluation | Teams wanting an open-source option”
“DeepEval | Code-based LLM evaluation and automated testing”
“DeepEval | Code-based LLM evaluation and automated testing”
“Ragas | RAG quality, relevance, and faithfulness evaluation”
“Ragas | RAG quality, relevance, and faithfulness evaluation”
“Galileo | AI quality evaluation and production monitoring”
“Galileo | AI quality evaluation and production monitoring”
“Datadog LLM Observability | AI monitoring integrated with conventional application observability”
“Datadog LLM Observability | AI monitoring integrated with conventional application observability”
“GitHub Actions for automated release gates.”
“LangSmith | Agent tracing, debugging, evaluations | LangChain/LangGraph and general agent development”
“LangSmith | Agent tracing, debugging, evaluations | LangChain/LangGraph and general agent development”
“Its strength is connecting development, evaluation, and production debugging in one workflow.”
“Arize Phoenix | OpenTelemetry-based tracing and evaluation | Teams wanting open-source infrastructure”
“Arize Phoenix — best for open, standards-based tracing”
“It can be self-hosted, making it attractive for teams with data residency or infrastructure requirements. Arize AI”
