Anyone starting a tech startup will face this decision: Should I start with a database that has built-in support for sharding? Or should I just start with MySQL or PostgreSQL and defer the sharding question for later?
Notion chose to do manual sharding (aka application-level sharding). That's what you end up doing if you didn't choose a database that has built-in sharding from the get go, because it is extremely hard to switch to a different database technology. (Larry Ellison compares it to a catholic marriage -- there is no divorce!)
I skimmed through the article to find the critical piece of info I was looking for: rationale for doing manual sharding. The rationale supplied in this article is "we wanted control over the distribution of our data." That's a weak explanation. It's the kind of thing you say to justify the bad choice made earlier on: failure to choose a database that supports automatic sharding from the get go.
> By mid-2020, it was clear that product usage would surpass the abilities of our trusty Postgres monolith, which had served us dutifully through five years and four orders of magnitude of growth.
This should answer your question. Just use a standard database and get on with coding features instead.
If you are successful enough to get to the point that sharding PostgreSQL becomes your bottleneck, you've won.
In addition, you really can't solve scaling problems up front. Where your bottleneck actually occurs will differ from where you think it will.
We will be happily building out features while your engineering team wastes time solving for scaling problems that you will never have because you don't have features your customers want.
Choosing database A instead of database B doesn't mean you're suddenly "solving for scaling problems". It just means you're better prepared to one day solve scaling problems, should your product take off.
Every choice has trade offs. And database sharding should be very low in your priorities when starting a project because YAGNI. In particular, I will actively laugh at anyone not choosing Postgres « because it does not scale enough » when starting a project. Postgres Pros overwhelm massively this imaginary potential Con.
Serious question: Let's say I'm using Django or Rails, and I just want to write the code with the ORM. What database am I using that you think makes sense?
I would see an argument that Notion's structure should have made this problem more important for them to resolve earlier in the process (like some document-based DB), but Postgres works well!
Probably CockroachDB - I'm not sure what other free/open source options there are.
I'm sure some proprietary databases will do it too but I haven't used Oracle/MSSQL/whatever to know what they give you.
What open source databases support automatic sharding? I think their only other option would have been something like cockroachdb, which probably wasn't very mature when notion started.
PostgreSQL supports native sharding out of the box. It's "manual" in that you have to set up table partitions and foreign data access yourself, but that's arguably still preferable to hacking together the whole thing as a pure application-layer concern.
It was my understanding that PostgreSQL supports table sharding out of the box on a single instance but not the ability to do distribute the processing across multiple instances.
Postgres supports FDW. Also you can use Aurora Postgres in AWS which can take you fairly far. Another option for many SaaS products you can simply run multiple clusters and assign
a given tenant to a particular cluster.
Notion chose to do manual sharding (aka application-level sharding). That's what you end up doing if you didn't choose a database that has built-in sharding from the get go, because it is extremely hard to switch to a different database technology. (Larry Ellison compares it to a catholic marriage -- there is no divorce!)
I skimmed through the article to find the critical piece of info I was looking for: rationale for doing manual sharding. The rationale supplied in this article is "we wanted control over the distribution of our data." That's a weak explanation. It's the kind of thing you say to justify the bad choice made earlier on: failure to choose a database that supports automatic sharding from the get go.