If anyone is looking for ideas for how to build tooling that fights flaky tests, I consolidated a number of lessons into a tool I open sourced a while ago.
It will do things like separate out different kinds of test failures (by error message and stacktrace) and then measure their individual rates of incidence.
You can also ask it to reproduce a specific failure in a tight loop and once it succeeds it will drop you into a debugger session so you can explore what's going on.
https://github.com/ajbouh/qa
It will do things like separate out different kinds of test failures (by error message and stacktrace) and then measure their individual rates of incidence.
You can also ask it to reproduce a specific failure in a tight loop and once it succeeds it will drop you into a debugger session so you can explore what's going on.
There are demo videos in the project highlighting these techniques. Here's one: https://asciinema.org/a/dhdetw07drgyz78yr66bm57va