If you're relying on backups for servers other than your database then you're keeping state on your servers and that's a Bad Thing. You should regularly destroy your own servers and recreate them using your configuration / deployment scripts if the prospect of this happening worries you. Do it before your business starts to rely on it.
For database servers you need to have procedures in place to quickly switch the production application over to a database you just spun up and migrated the data from a recent backup to in order to test your data backups. You want this to happen as smoothly as possible in the case of a failure. Keep data backups in three different places and test your procedure on all of them.
On a production system, do not keep the database on the same server as the application.
It's so insanely easy to get started with puppet or ansible I wouldn't start rolling out a product without using them from the get-go at this point. At the very least properly package your software (read: DEB or RPM, not docker containers) so it's easy to reinstall quickly if need be.
Personally, I like platform libraries rather than OS packages. I have several Ruby projects set up as gems. What's neat about that is it allows code reuse across projects.
So go ahead and vendor them in, outside of inclusion in distribution software packages (with exceptions) there's nothing stopping you from including your own dependencies. Packaging your application properly makes deployment a hell of a lot easier and less error-prone.
And the nice thing about DO and some of their cohort is that they charge by the hour for servers. Fire up new droplets and install your upgrades on them, smoke test the system, swap your routing over to the new machines, back up the old droplets (just in case you missed something) and and then destroy them.
If that takes 4 hours you've hardly spent any money at all, and you're running your cluster the way a lot of people already do.
Because state left on servers is inevitably unmanaged and will get lost eventually. At my job, I have cron jobs running on the production system from years ago that I have no clue what they do and no time to try to figure it out. I find out when they fail and someone, usually customer service, complains.
If the server goes away, then even once I redeploy, I've lost all those cron jobs. Who knows what will happen then.
On systems I build, cron jobs are added as part of the deployment process and managed as code, in the git repository.
That's just one example. Others include iptables rules, FTP server configuration, startup scripts, and such. If it's not in your codebase, it's unmanaged state on the server that you will lose if your cloud provider takes a shit. All of that needs to be managed as code or as configuration and deploy scripts should contain idempotent commands that add them if they are not present. I consider shelling into a server for any reason to be unideal and will think of ways to avoid it.
If you are asking about database being on the same server as the application, it's because it makes infrastructure management more difficult. Application servers are set up quickly, torn down quickly, and work as soon as you deploy. Database servers are fortresses, you don't build new ones and tear them down half as quickly, unless you've got scaling requirements.
It is tempting to want to put data on the same server as the application early on in order to save on hosting costs, with the understanding that you'll separate them later, but that's a rookie move. It's much less work to separate them now, when there's nothing riding on it, much less headache.
The other reason to keep the DB separate is that they have different IO patterns that are not always complementary, and it's easier to gather diagnostic metrics if the data isn't munged. In the days of physical servers you'd also choose different hardware for the DB machine.
Configuration capture is a big thing, and it separates the adults from the kids. I've had hardware failure of my dev box twice in my career, and when I tell certain people that I have to rebuild my machine you can see their faces blanche. These are the people I know I haven't converted yet.
Everything important, except for the handful of things I'm actively investigating, is always stored on shared machines. Version control. Wiki. Some sort of artifact repository. The instructions for getting an environment set up with version X of our software are also stored in one of those places, and have been vetted by every new developer and some of the QA team. I can have my machine back up and running in a couple of hours, and most of that is waiting for downloads, if we didn't have the presence to store copies locally.
Why do I care about this stuff? Seems sort of OCD on the face of it, and maybe you're right. But sooner or later you're going to have a high severity issue in production that should be an all-hands-on-deck affair, and if you haven't done this work, losing a hard drive on a dev laptop will be the least of your worries. Everyone busy doing work on X+1 needs to be able to get back to version X and all of its dependencies in under an hour, and by themselves, because the people who could help them are most likely on the front lines of fixing the bug as fast as possible.
What's more, someone probably needs to get versions X-1 and X-2 running to figure out if you need to warn people using older versions. So that's getting people running 3 or 4 versions of your software autonomously, so that you can identify repro steps, long and short term mitigation strategies, verify that they work, formulate a bulletin and provide patches for people. Not only do you need to get your configuration captured/documented, you need it captured 2-3 versions of your software ago, which means you need to start thinking about this stuff pretty early in your project.
I think the other point of contention was your term "database". There are lots and lots of systems out there that rely on data that is not stored in a database. The data is the same kinds of things you would store in a databases but they exist as files on a filesystem instead (or in another data storage devices that is usually not referred to as a database).
That said you could easily use the same advice for that data. But in some cases the application itself needs to be on the same server as the database. Think if you were building postgres for instance...
> There are lots and lots of systems out there that rely on data that is not stored in a database. The data is the same kinds of things you would store in a databases but they exist as files on a filesystem instead (or in another data storage devices that is usually not referred to as a database).
If you can't afford to lose it, then it needs to be appropriately managed so it can be recreated in the case that it's lost. If you can afford to lose it, then it's not really state and you can forget about it.
I think people will understand more easily if instead of saying "database" specifically, you say "database tier."
I'm pretty sure the usual understanding of the terms "business tier" and "database tier" is that the "business tier" is a bunch of ephemeral nodes, and the "database tier" is where-ever the data from the "business tier" goes to be persisted.
For database servers you need to have procedures in place to quickly switch the production application over to a database you just spun up and migrated the data from a recent backup to in order to test your data backups. You want this to happen as smoothly as possible in the case of a failure. Keep data backups in three different places and test your procedure on all of them.
On a production system, do not keep the database on the same server as the application.