Yeah, what I mentioned does nothing to improve performance, just thought I'd mention it as something to think about. For your use case S3 replication doesn't really give you extra availability. In practice I would likely use your caching method, and use S3 replication in addition, if I had a large dataset that doesn't cache well. Or if I absolutely needed to maintain write availability, in which case I would use bi-directional S3 replication.
Definitely. If our dataset was absolutely massive and we couldn't hold a reasonable amount on disk, it'd make more sense. Fortunately, we are getting a 90+% hit ratio out of a very small amount of space relative to the size of our bucket.
And yeah, we have 0 resiliance for write data here. Again, fortunately, we can afford this tradeoff since the amount of uploads is significantly lower and much less critical for us.
