-
Notifications
You must be signed in to change notification settings - Fork 77
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Migrate docs.rs to RDS and ECS #353
Comments
another thing we need to figure out:
|
after checking our NGINX config there is a second piece we need to solve somehow: IP blocks. Every now and then we have a mis-acting crawler and in these cases we blocked the source IP in NGINX on our server. I would prefer to have this in AWS / CloudFront if possible. Otherwise we would add this to our web container, probably configured via environment variable? |
Next piece we need before prod : Access to logs |
For blocking IPs, we should just set up a web-application firewall (AWS WAF). I actually think that we already have one set up for docs.rs, but I'm not 100% sure. Access to the logs is a good point! It probably makes sense to stream all logs to a central place, whether that's CloudWatch or an external tool like Datadog. |
@jdno Please let me know if you need a hand with any of the items in this list 🙂 |
@jdno coming from this discussion I want to add here that the docs.rs containers / servers should not be reachable directly from the internet. So all traffic needs to go through CloudFront & AWS WAF |
One thought I had thinking about this topic again:
from rust-lang/docs.rs#1871 (comment)
I'm not sure if we can / should handle invalidations differently, but we might think about using fastly when we rework the infra? |
Can't we de-duplicate invalidations if we approach the limit? E.g., a * invalidation every 5 minutes would presumably never hit the limit. Not sure how that would affect cache hit rates, but I'd expect designing around not needing invalidations or being ok with fairly blanket invalidations to be a good long-term strategy. (I think we've had this conversation elsewhere before). |
You mean "escalating" them, so when the queue is too long, just convert the queue into a full purge.
This also means that the backend always has to be capable to handle the full uncached load, and higher egress costs depending on how often we have to do the full purge. I also remember a discussion at EuroRust that we could think about having additional docs.rs webservers (also readonly DB & local bucket?) in some regions (europe?).
You're right, this is a valid discussion to have. I imagine this would only work when the URLs would include something like the build-number in the URL, and replace the more generic URLs rest with redirects. If I'm not missing something this would revert some of the SEO & URL work from rust-lang/docs.rs#1438 (introducing
you're probably right :) I wanted to bring it up here as a point, for when we migrate infra anyways. |
Note that (IMO) if we can get the cache keys setup right, i.e. everything except HTML is always at a by-hash file path - it seems to me that /latest/ can just be served with a short ttl (5 minutes), perhaps with stale-while-revalidate. That means that there's a small period where it's not necessarily consistent what version you get from it across all pages if some are cached locally and some aren't (and likewise for CDN), but I don't see any real problem with that. Users mostly won't even notice. Yes, especially anything out of s3 can definitely be replicated if we need it to be into multiple regions pretty easily. This just causes issues while you need invalidations since you're racing against replication which can itself take some time (hours IIRC for the cheap option and minutes for the costly one?) |
Jep, everything except HTML should have already have hashed filenames, with some small exceptions.
That's good to know, thanks! |
Questions
The text was updated successfully, but these errors were encountered: