This kind of stuff I like reading on HN!

The link throws a 503 rn. I wonder what would be the reason

Sorry, autoscaling took a second to ramp up

Initially thought only Zoom is having issues so posted with zoom status. But it was widespread outage. Zoom seems to have recovered by now. Also, Zoom has a variety of hosting + colo DCs at a lot of places.

Looks like a lot of services are impacted including Cloudflare, Ping, Zoom and Datadog.

Not sure why we're on that list. We run our own infrastructure and are not built on AWS.

Looks like Snap, Crackle and Pop are down as well.

> Looks like Snap, Crackle and Pop are down as well.

I don't work on cloud stuff, so I'm genuinely unsure if this is a joke.

It's a joke but I only knew that because Snap is/was (as of S1) hosted on GCP and not AWS. Crackle happens to be the name of a video on demand company.

It's a reference to the breakfast cereal of the same name.

Pop is also a screen sharing/pairing tool, so the joke was great.

Pedantic clarification for the unfamiliar: the breakfast cereal is named Rice Krispies while Snap, Crackle, and Pop are the names of the cartoon mascots on the box.

Hah, frequency illusion strikes again? I just learned about the derivatives past Jerk yesterday.

Cloudflare uses AWS? For what?

AWS acknowledged the issue https://health.aws.amazon.com/health/status

[10:25 AM PDT] We can confirm that some instances within a single Availability Zone (USE2-AZ1) in the US-EAST-2 Region have experienced a loss of power. The loss of power is affecting part of a single data center within the affected Availability Zone. Power has been restored to the affected facility and at this stage the majority of the affected EC2 instances have recovered. We expect to recover the vast majority of EC2 instances within the next hour. For customers that need immediate recovery, we recommend failing away from the affected Availability Zone as other Availability Zones are not affected by this issue.

Funny I failed away from the zone and RDS still doesn't work, connections fail.

Edit: 2 minutes after I post this it starts working.

"[10:11 AM PDT] We are investigating network connectivity issues for some instances and increased error rates and latencies for the EC2 APIs within the US-EAST-2 Region."

yeah looks like us-east-2 has networking issues

Zoom is having connectivity issues. Nothing on https://status.zoom.us/ page yet.

Why were all BGP routes advertised from same set of servers in same DC which were pointed by ALL fb owned domains?

I don't think they were. But, without knowing more specifically what happened, it is hard to speculate.

Some of the speculation I have seen was that this was an attempt to cut connections to one internet exchange (IXP) point, possibly resulting in cutting connection to all IXPs. But ,as I said, this is speculation. It'd take a more thorough understanding of how the automation looked prior to the incident, the change hat was trying to be made, and eth like, to say something sensible.

I just heard from a contact that the fallback/backup IRC is also down.

Bet it was located at irc.facebook.com ;)

Joking aside, I can see how an IRC network has potential to be used in these situations. Maybe FAMANG should work together to set something like this up. The problem is, a single IRC server is not fail safe, but a network of multiple servers would just see a netsplit, in which case users would switch servers.

Also, I remember back in the IRCnet days using simply telnet to connect to IRCnet just for fun and sending messages, so its a very easy protocol that can be understood in a global desaster scenario (just the PING replys where annoying in telnet).

I heard the same thing from my old coworker who is at FB currently. All of their internal DNS/logins are broken atm so nobody can reach the IRC server. I bet this will spur some internal changes at FB in terms of how to separate their DR systems in the case of an actual disaster.

it is always DNS!