I've been the victim of this, a producer consumer system with a single index file using some variant of a heap.
For each incoming request the producer;
locked the file, read the heap from the file, added the value to the heap, then wrote it back and unlocked it.
For each consumer thread, the thread did the reverse. It was setup with threads created by incoming requests, and a threadpool of 128 threads for the consumer.
This worked seamlessly until one day, at 3am PST, a flood of requests came in and the index file grew to ~4mb, and everything ground to a halt.
The legend of the architecture was a senior engineer did this on a weekend, then a month later left the company.
It isn't about making an architectural design decision / change at 3am, but someone has to live with the consequences of such a design decision and fix something in some way or another at 3am in the morning.
I.e. you could go with a clustered db setup which is supposedly HA, or a classical single instance DB...
The point here is, whoever has to get up at 3am to fix a potential catastrophic failure in the setup should have a say in it.
If the architect is also the coder/support, then it's a moot point. But if there is a separate architect, or a pack of them, there's no point in waking them up at 3am. Any architectural change isn't going to be deployed within the hour.
Personally, I think the designers/architects should work on the code too. You have to dogfood your own design, so to speak.
When does a call at 3am involve an architectural decision or change? It should never come that far.