Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I found myself doing very much this my very first day on the job working for a software startup.

We had a Grails app that acted as a front end for a number of common DB interactions, which were selected via a drop down. One of these (in fact, the default) action was titled "init DB". Of course, this would drop any existing database and initialize a new one.

When running through the operational workflow with our COO on the largest production database we had, I found myself sleepily clicking through the menu options without changing the default value. I vividly remember the out of body experience the OP describes, and in fact offered to fire myself on the spot shortly thereafter.

It's fun to laugh about in hindsight, but utterly terrifying in the moment - to say nothing of the highly destructive impact it had on my self confidence.



How did the company deal with the loss of that database? Did they actually have backups, and just restored the data? Did they reconstruct the data from other sources?


In our case we had periodic backups, and together with filesystem logs were able to restore most of the data. However, we were hosting highly sensitive data and the work being done was time critical. The downtime was therefore not popular with our clients, who were losing ~$15k per hour offline.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: