“Yes. Braintrust can automatically run evaluation suites in CI/CD on every pull request and catch prompt regressions before code is merged.”
“Braintrust can automatically run evaluation suites in CI/CD on every pull request and catch prompt regressions before code is merged.”
“It supports this through a dedicated GitHub Actions integration, braintrustdata/eval-action@v2, as well as a CLI (bt eval) that works with other CI systems.”
“The important distinction is that Braintrust doesn't just run tests: it can compare evaluation results against a known-good baseline and enforce quality thresholds.”
“Regression detection: Identify individual test cases whose scores deteriorate.”
“PR reporting: Display evaluation improvements and regressions directly in GitHub.”
“Experiment history: Keep inputs, outputs, scores, and traces for debugging.”
“Braintrust supports custom reporters for this.”
“Braintrust can serve as an automated quality gate for LLM applications, much like unit tests and integration tests do for traditional software.”
“One important implementation detail: reporting a regression does not automatically block a merge.”