This is why it's important to isolate production from other environments. Three ... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		donw on Nov 15, 2010 \| parent \| context \| favorite \| on: Cause of today's Github outage This is why it's important to isolate production from other environments. Three rules have kept me from ever borking a production database: 1. Production DB credentials are only stored on the production appservers, and copied in at deploy time. 2. The production DB can only be accessed from the IPs of the production webfarm. 3. Staging, Testing, Development, and Everything Else live on separate networks and machines than production.

oddi on Nov 15, 2010 [–]

While this is part of the problem, to me it seems like they didn't have proper restore procedures or at least hadn't tested them enough. There are countless ways to corrupt a database and restoring from backup is part of most remedies.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact