Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

The gist of this seems to be that DynamoDB becomes a problem if you have millions of customers.

Don't worry. You don't. And there will be many good reasons to refactor the architecture before you do.



We have close to 25M monthly active users and DynamoDB works pretty well for our use case.

We store user generated content in DynamoDB. Our largest table is over 1 TB. The mapping of users to their generated content is in Postgres. So doing work on behalf of a particular user will generally be distributed across multiple nodes.

We've enjoyed the very predictable performance of DynamoDB as well as the operational simplicity. We've started moving smaller datasets over and are using it more and more.


How large is user's generated content in average? Have you considered S3 for your use case? Storage in DDB can be quite pricey (when combining storage cost and and read capacity usage cost)


We are a messaging application, so every chat message is stored in DynamoDB in an ordered stream. We frequently fetch slices of a stream, so the granularity of storage is a single message.

S3 does not do well (cost wise) with billions of files. Provisioning 1000 Write IOPS on Dynamo costs around $100/month. That's 2.5B writes. On S3 that's going to be on the order of $10k. Similarly 1000 Read IOPS on Dynamo is ~$20/month vs 2.5B reads on s3 costing $1000. Storage is cheaper on S3 by about 10x, but per TB that comes out to $250 vs $25, which is hardly a dominating cost factor.


Sounds interesting! But if the mapping is stored in Postgres, your system's availability is capped by that.


Yes. We're migrating towards a more available store (one option we're testing out is Aurora) but availability wasn't our primary motivation in choosing DynamoDB. The main motivation was consistent performance with a dataset too large to fit into memory.


I often give this same advice, but assuming this is the case, why reach for DynamoDB at all? Are there small-data use cases where DynamoDB makes more sense than RDS or a hosted redis? At least with those, if you do run into scaling issues you haven't locked yourself into dynamo.


I would imagine the use of AWS Lambda might be a pretty decent incentive to use DynamoDB. I'm guessing that stateless DynamoDB queries are faster to fire off than dealing with Postgres or MySQL connection setup and teardown.


I'm not familiarized with DynamoDB, but why would connecting to it be any faster than connecting to Postgres or MySQL?


Perhaps DynamoDB is based around HTTP, rather then requiring socket back and forth actions(uname + pass, auth, query, results vs just a single hit where query + auth details are packed together). Never actually looked at dynamodb, so not sure.


Yes, that is accurate. DB operations to DynamoDB are just AWS API calls, the same as you would use to spin up an EC2 instance or create an S3 bucket.


Lambda keeps the containers running if there's traffic coming to them, so the setup cost isn't paid per request unless there are few requests.


I'm not sure how much this applies to dynamodb.. I think I couldn't even figure out how to get started with that.

I used simpledb once as part of project. All I needed it for was to store a small amount of state to be shared between some otherwise stateless ec2 instances. So, kind of the perfect use case.. Except I don't think it is made to scale down that small.

I don't know if I provisioned the capacity wrong or what, but it was so flaky. All I needed it to do was a few reads every few minutes.

I think my issue was I had the provisioned scalability set down to the smallest value I could because I only needed to pull down ~10 rows every 60 seconds or so.. But when the app would start and try to sync up, simpledb would throttle it?

I ended up having to just write a cron job to sync the db to a local json file, because doing the reads from inside my main loop would randomly fail or timeout.


For my very small-data use case I would be very happy if I could use DynamoDB as a persistent data store because the operational aspect is like ten lines of Terraform


Well, I just spent the past month dealing with issues trying to get around 360M records into Dynamo... At this point, with the number of partitions, I'm unsure if we'll be able to keep reads to a decent level without bumping the throughput to much higher than what's needed.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: