The postmortem will should interesting, can't imagine how even just basic integration testing didn't catch this. Much less basic best practice like canarying.