There are so many other possible factors: timings, interactions with other processes, memory allocations, network conditions, etc. A random seed is just one of several non-deterministic factors that could have been the cause.
Fair enough, though interactions with other processes and network conditions are not relevant for unit tests. Timings maybe, but then you still need a strategy to turn a failing test into information you can use to fix a bug.
I have horrible memories of time spent tracking down other test failures caused by helper programs not shutting down cleanly, and causing obscure bugs in later tests. e.g. a test failing because it couldn’t bind to port 443, because an earlier test helper program sometimes didn’t clean up properly. That’s just one possibility for a process and/or network condition that ruins a test.