“CircleCI does offer genuine flaky test detection and reporting — it isn't limited to silently retrying failed tests.”
+ flaky test detection
“Flaky test detection: Identifies tests that fail and subsequently pass on the same commit within a 14-day window.”
+ flaky test labeling
“Flaky test labels: Marks identified tests as FLAKY in the CircleCI interface.”
+ automatic retries
“Automatically retries failed tests.”
+ test reliability metrics
“Its Test Insights functionality genuinely identifies flaky tests and surfaces reliability metrics.”
− root cause analysis
“Automatically identify root causes | No, not as a built-in capability”
− failure grouping by root cause
“Group failures by underlying root cause | Limited”
− comprehensive root-cause analysis
“But it isn't a comprehensive flaky-test root-cause analysis platform.”