bitswap/httpnet: request with CDN-Loop (RFC 8586)

[lidel]

We need to find a solution to situation where defunct gateways start returning redirects back to ipfs.io and dweb.link, effectively acting as amplification vector.

There is some prior art solution in form of CDN-Loop header which helps in avoiding unintended HTTP loops like that:

This document defines the CDN-Loop request header field for HTTP.
CDN-Loop addresses an operational need that occurs when an HTTP
request is intentionally forwarded between Content Delivery Networks
(CDNs), but is then accidentally or maliciously re-routed back into
the original CDN causing a non-terminating loop. The new header
field can be used to identify the error and terminate the loop.

Ref.

• https://datatracker.ietf.org/doc/html/rfc8586
• https://blog.cloudflare.com/preventing-request-loops-using-cdn-loop/
• gateway: CDN-Loop (RFC 8586) specs#426

I think in case of Rainbow, we could leverage SEED and have all nodes set CDN-Loop to the same deterministic hash value, and refuse to process requests with matching one. This way HTTP traffic looped/proxied back to our infra is ignored.

cc @hsanjuan for feedback, maybe there a better way?

[hsanjuan]

@lidel we could simply not support redirects... Our urls come from content provider records, which, iirc, have expirations and need to be renewed regularly. If an endpoint changes, the action to take is to update the provider record, rather than setting a redirect.

It's also not even about loops, its also about redirect hops (1 hop means opening a new connection somewhere else which is already expensive by itself in our context).

[lidel]

@hsanjuan good point. not following redirects sounds ok to me.

interpreting 3XX as 404 will force people to update announcements rather than keeping dead domains alive and then breaking months/years later.
take a look at proposed spec clarification in ipfs/specs#494

still, the utility of CDN-Loop goes beyond redirects. we've seen people setting up dumb Nginx-based proxies that blindly forwarded (not redirected) requests to ipfs.io with all original headers (cough rogue Rhea/Saturn nodes).

perhaps for now, disable following redirects, and keep this issue about CDN-Loop as low priority (P3) until we actually need it?