You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
HTTP 302 Found responses with Cache Headers not being cached #384
The particular use case for this problem is the JFrog Artifactory uses 302 Found responses for signed URLs to cloud storage in HTTP requests to their PYPI proxy cache. Python PIP which leverages the CacheControl library in turn fails to cache those responses and downloads the packages everytime. This is a pain for CI processes which download large packages like PyTorch.
Additionally RFC2616 section 10.3.3 claims the 302 Found HTTP response can be cached if the Cache-Control or Expires response headers are present.
I think the reason we don't cache this is because the various RFCs are ambiguous about temporary redirects vs. permanent ones: RFC 2616 doesn't proscribe caching for these ("cacheable" isn't qualified by SHOULD or similar), while RFC 7231 doesn't mention caching for 302 at all and RFC 7234 mentions only permanent redirects explicitly.
I'm not opposed to changing this per se, but I'd like to better understand why JFrog is using HTTP 302 instead of HTTP 301 for these redirects -- ideally this is something they'd fix on their own end, since other mirror indices don't appear to have this problem.
(More broadly, I agree with the appraisal in pypa/pip#13396 (comment) -- it would be good to better understand the intended behavior here, since a signed URL strongly suggests that caching shouldn't be done.)
As you pointed out the standards are vague on required behavior, so I interpretted it as a suggestion as well. I do not know how common the usage of a private CDN redirect is, so I thought to bring it up and see if there is a best practice.
I will take these two tickets to JFROG and have them advocate for a behavior or have them alter theirs. Thank you.
@woodruffw I believe JFrog is not using HTTP 301 for redirects likely the 'Location:' of the redirect has same url but different query component, i.e., the signature, e.g., X-Goog-Signature is different each time?
Not caching the initial request that results in the HTTP 302 redirect may not be that critical since the relevant content returned is tiny with just the Location: being important.
Would the concern be more of NOT caching the redirected Location: url which is the larger piece of content being transferred. I did some tests, and is it correct to say that pip performs a cache hit/miss based on the entire url including the query components? This results in the caching NOT happening because each time the X-Goog-Signature changes?
I believe JFrog is not using HTTP 301 for redirects likely the 'Location:' of the redirect has same url but different query component, i.e., the signature, e.g., X-Goog-Signature is different each time?
Sorry, could you clarify what you mean here? "A same URL but different query component" is not the same URL, since the query is part of the URL. I'm guessing you mean "same path component," but I want to make sure I understand correctly.
I did some tests, and is it correct to say that pip performs a cache hit/miss based on the entire url including the query components? This results in the caching NOT happening because each time the X-Goog-Signature changes?
I think the pip maintainers would better be able to answer what pip specifically does -- all I know is what the caching middleware does, which pip may or may not tweak, do a superset of, etc.
For CacheControl specifically, we (currently) key off of the query string exactly as it appears in the request via _urlnorm here:
@woodruffw Sorry that I was not clear with what I meant by "A same URL but different query component". I have an AWS cloudfront/s3 pre-signed url on hand so I will use that as an example.
The 'main' url is https://d3njxchdk64paq.cloudfront.net/development/2/7/pypi/packages/8.whl and this does not change for a pre-signed url. Then there is the query components/url parameters, i.e., everything after ?, e.g., Expires, Signature, etc., which usually changes each time the 'main' url is pre-signed.
Thank you for clarifying with:
(a) This quote "... the query is part of the URL"
(b) The code snippet showing (a)
I think this is the crux of the discussion (I hope I didn't misinterpret anyone. At the very least our users (Buildkite/Packagecloud) are seeing the same phenomenon), i.e., each pre-signed url (whether google storage, aws cloudfront/s3, etc.) if considered with query component/string, is always different even though the pre-signed url is pointing to the same object in the bucket, thus cache hit never happens. This is because the pre-signed url contains query component/string with Expires and Signature (for aws) and similar counterparts for other cloud, that is always changing each time a pre-signed url is generated.
uv, which is a pip alternative on the other hand behaves as expected, i.e., performing cache hit/miss based on just the url without considering the query components/strings.
If you can help point to any RFCs that indicates cache hit/miss should take the full url including query components/string, it will be super helpful to determine where we can fix this issue.
If you can help point to any RFCs that indicates cache hit/miss should take the full url including query components/string, it will be super helpful to determine where we can fix this issue.
And to my read, the requirement here is to include the query component in the "effective request URI" (since it's a semantically relevant part of the URI).
I don't know exactly how uv behaves in this regard, but if you're seeing cache hits on signed URLs that vary by query component then I suspect that something, somewhere, is violating cache-control semantics. That, or there's some special casing around these kinds of signed URLs, since they're uncacheable by design.
However, to take a step back: can you confirm that what you're seeing is related to the original issue here? Specifically, is your first URL performing a 301 redirect or a 302 redirect?
(Another possibility is that uv is doing application level catching instead of HTTP response caching. That would sidestep the problem entirely, but I don't know if that's actually what's happening.)
I looked into this, and I don't believe uv is doing application-level caching -- it has HTTP caching middleware that (like CacheControl) doesn't cache the HTTP 302 redirect. As far as I can tell uv's caching should also invalidate on URL query changes (there are some places in uv where the query component is removed, but FWICT these don't affect the cache key).
However, to take a step back: can you confirm that what you're seeing is related to the original issue here? Specifically, is your first URL performing a 301 redirect or a 302 redirect?
This is what pip is doing when I did pip install verysimplemodule===0.0.1 the second time (I deleted the package after first install but left the cache untouched)
[A] pip request for simple index
[B] pip gets 302 redirect
[C] pip says 302 is NOT in (200, 203, 300, 301, 308)
[D] pip looks up the redirect Location: header which points to AWS cloudfront pre-signed url (which has Expires query parameter that is constantly changing each this the same AWS cloudfront url is pre-signed
[E] pip found not cache hit for the AWS cloudfront pre-signed url as it considers all query parameters when searching for cache hit
[F] pip downloads the python package from AWS cloudfront pre-signed url
2025-07-14T06:07:35,373 Found index url http://packagecloud.localhost/khor/another-repo/pypi/simple/ <====== [A]
2025-07-14T06:07:35,373 Looking up "http://packagecloud.localhost/khor/another-repo/pypi/packages/verysimplemodule-0.0.1-py3-none-any.whl" in the cache
2025-07-14T06:07:35,373 No cache entry available
2025-07-14T06:07:35,373 No cache entry available
2025-07-14T06:07:35,403 http://packagecloud.localhost:80 "GET /khor/another-repo/pypi/packages/verysimplemodule-0.0.1-py3-none-any.whl HTTP/1.1" 302 0 <======== [B]
2025-07-14T06:07:35,403 Status code 302 not in (200, 203, 300, 301, 308) <======== [C]
2025-07-14T06:07:35,404 Looking up "https://d3njxchdk64paq.cloudfront.net/development/2/7/pypi/packages/8.whl?Expires=1752474000&Signature=qORpa8TJo3OgXQRdI3ASkvvwQ5OVNBhSX1kUQjkJ964CvYsDxLXr8Dg8w0PraCRchLOP1fw-bO1UWEC7xiHGRYHYvMo27u0mCfnZDhf~q-sxEO9uZdK-Q7QNdRUBGoiveYB87g9HgzKh3G0Im8TuRmdyhpOFXOgdWlZK1JgDZG9vPOvynJjGzazkuIb9S~Wj8YfcTy3ul7hDzIJpfi24VeIP9fG3O9rmQM8cZ1SRmAgPZeC5Y3-zXSn9Hpu2bT346CW7uhmoSdGyqSAcUewZ1vCpo-UpxiJvg~TQ9KkAuYJsVFxFuJY9~bsvkqgLQ82Kad31w-i~bOb6Ym7nnSAJnw__&Key-Pair-Id=K1126YPJP5H8P0" in the cache <======== [D]
2025-07-14T06:07:35,404 No cache entry available <======== [E]
2025-07-14T06:07:35,404 No cache entry available
2025-07-14T06:07:35,876 https://d3njxchdk64paq.cloudfront.net:443 "GET /development/2/7/pypi/packages/8.whl?Expires=1752474000&Signature=pMRpa8TJo3OgXQRdI3ASkvvwQ5OVNBhSX1kUQjkJ964CvYsDxLXr8Dg8w0PraCRchLOP1fw-bO1UWEC7xiHGRYHYvMo27u0mCfnZDhf~q-sxEO9uYzK-Q7QNdRUBGoiveYB87g9HgzKh3G0Im8TuRmdyhpOFXOgdWlZK1JgDZG9vPOvynJjGzazkuIb9S~Wj8YfcTy3ul7hDzIJpfi24VeIP9fG3O9rmQM8cZ1SRmAgPZeC5Y3-zXSn9Hpu2bT346CW7uhmoSdGyqSAcUewZ1vCpo-UpxiJvg~TQ9KkAuYJsVFxFuJY9~bsvkqgLQ82Kad31w-i~bOb6Ym7nnSAJnw__&Key-Pair-Id=K1126YPJP5H8P0 HTTP/1.1" 200 1242
2025-07-14T06:07:35,877 Downloading http://packagecloud.localhost/khor/another-repo/pypi/packages/verysimplemodule-0.0.1-py3-none-any.whl (1.2 kB) <======== [F]
Likewise when using uv to download verysimplemodule===0.0.1 the second time (I deleted the package after first install but left the cache untouched):
[A] uv request for simple index
[B] uv seems to have found the cached url for simple index, and requested revalidation
[C] uv got modified response for simple index
[D] uv retrieved the newer simple index
[E] uv conclude that there is a cache hit for the python package so no need for redownload
DEBUG Found stale response for: http://packagecloud.localhost/khor/another-repo/pypi/simple/verysimplemodule/ <===== [A]
DEBUG Sending revalidation request for: http://packagecloud.localhost/khor/another-repo/pypi/simple/verysimplemodule/ <===== [B]
DEBUG Found modified response for: http://packagecloud.localhost/khor/another-repo/pypi/simple/verysimplemodule/ <===== [C]
DEBUG Searching for a compatible version of verysimplemodule (>=0.0.1, <0.0.1+)
DEBUG Selecting: verysimplemodule==0.0.1 [compatible] (verysimplemodule-0.0.1-py3-none-any.whl)
DEBUG Found fresh response for: http://packagecloud.localhost/khor/another-repo/pypi/packages/verysimplemodule-0.0.1-py3-none-any.whl.metadata <===== [D]
DEBUG Tried 2 versions: test 1, verysimplemodule 1
DEBUG all marker environments resolution took 0.408s
Resolved 2 packages in 409ms
DEBUG Using request timeout of 30s
DEBUG Registry requirement already cached: verysimplemodule==0.0.1 <===== [E]
Installed 1 package in 2ms
+ verysimplemodule==0.0.1
uv, which is a pip alternative on the other hand behaves as expected, i.e., performing cache hit/miss based on just the url without considering the query components/strings.
I wasn't aware uv cached HTTP responses at all. I know it caches built packages and other application specific artifacts.
Can you point to what you mean in the uv codebase, prior art of other applications implementing your intended behavior is useful to make informed choices.
This is probably obvious to everyone in the discussion but cache control is far more of a general middleware than pip's specific use case, and using http cache as the primary mechanism for catching is probably not perfect for pip, but cache control shouldn't be changed for that purpose.
I'm going to privately ping the uv team in case they have any input on the question of query components.
This is probably obvious to everyone in the discussion but cache control is far more of a general middleware than pip's specific use case, and using http cache as the primary mechanism for catching is probably not perfect for pip, but cache control shouldn't be changed for that purpose.
Yep, agreed -- I also think (barring more evidence) that what CacheControl and pip are doing here is probably more correct, since the observed states (302 redirects and changing queries) are straightforwardly not cacheable at the HTTP layer. But to your point, this is a discrepancy between what pip semantically wants to cache (application-level contents) and what HTTP caching actually enables (responses).
I'm going to privately ping the uv team in case they have any input on the question of query components.
@woodruffw Thanks for recommending running uv with TRACE, i.e., -vvv. Would love you hear your interpretation of it.
(test) root@orbstack:/test# uv -vvv --no-config --allow-insecure-host packagecloud.localhost add --cache-dir ./.cache --index=http://packagecloud.localhost/khor/another-repo/pypi/simple very
simplemodule==0.0.1
DEBUG uv 0.8.3
DEBUG Found project root: `/test`
DEBUG No workspace root found, using project root
TRACE Checking lock for `/test` at `/tmp/uv-d01170fa7b5d04fb.lock`
DEBUG Acquired lock for `/test`
DEBUG Ignoring Python version file at `.python-version` due to `--no-config`
DEBUG Using Python request `>=3.11` from `requires-python` metadata
DEBUG Checking for Python environment at: `.venv`
TRACE Found cached interpreter info for Python 3.11.13, skipping query of: .venv/bin/python3
DEBUG The project environment's Python version satisfies the request: `Python >=3.11`
TRACE The project environment's Python version meets the Python requirement: `>=3.11`
TRACE The virtual environment's Python interpreter meets the Python preference: `prefer managed`
DEBUG Released lock at `/tmp/uv-d01170fa7b5d04fb.lock`
TRACE Checking lock for `.venv` at `.venv/.lock`
DEBUG Acquired lock for `.venv`
DEBUG Using request timeout of 30s
DEBUG Found static `pyproject.toml` for: test @ file:///test
DEBUG No workspace root found, using project root
DEBUG Ignoring existing lockfile due to mismatched requirements for: `test==0.1.0`
Requested: {Requirement { name: PackageName("verysimplemodule"), extras: [], groups: [], marker: true, source: Registry { specifier: VersionSpecifiers([VersionSpecifier { operator: Equal, v
ersion: "0.0.1" }]), index: None, conflict: None }, origin: None }}
Existing: {}
TRACE Performing lookahead for test @ file:///test
DEBUG Solving with installed Python version: 3.11.13
DEBUG Solving with target Python version: >=3.11
TRACE Assigned packages:
TRACE Chose package for decision: root. remaining choices:
DEBUG Adding direct dependency: test*
INFO add_decision: Id::<PubGrubPackage>(0) @ 0a0.dev0 without checking dependencies
TRACE Assigned packages: root==0a0.dev0
TRACE Chose package for decision: test. remaining choices:
DEBUG Searching for a compatible version of test @ file:///test (*)
DEBUG Adding direct dependency: verysimplemodule>=0.0.1, <0.0.1+
INFO add_decision: Id::<PubGrubPackage>(1) @ 0.1.0 without checking dependencies
TRACE Assigned packages: root==0a0.dev0, test==0.1.0
TRACE Chose package for decision: verysimplemodule. remaining choices:
TRACE Fetching metadata for verysimplemodule from http://packagecloud.localhost/khor/another-repo/pypi/simple/verysimplemodule/
TRACE Cached request http://packagecloud.localhost/khor/another-repo/pypi/simple/verysimplemodule/ is storable because this is a shared cache and its response has a 'private' cache-control di
rective
TRACE Freshness lifetime found via cache-control max age setting: 0ns
TRACE Cached request http://packagecloud.localhost/khor/another-repo/pypi/simple/verysimplemodule/ has a cached response that does not permit staleness because the response has a 'must-revalidate' cache-control directive set
TRACE Request http://packagecloud.localhost/khor/another-repo/pypi/simple/verysimplemodule/ does not have a fresh cache because its age is 103 seconds, it is greater than the freshness lifetime of 0 seconds and stale cached responses are not allowed
DEBUG Found stale response for: http://packagecloud.localhost/khor/another-repo/pypi/simple/verysimplemodule/
DEBUG Sending revalidation request for: http://packagecloud.localhost/khor/another-repo/pypi/simple/verysimplemodule/
TRACE Handling request for http://packagecloud.localhost/khor/another-repo/pypi/simple/verysimplemodule/ with authentication policy auto
TRACE Request for http://packagecloud.localhost/khor/another-repo/pypi/simple/verysimplemodule/ is unauthenticated, checking cache 18:35:38 [6/1849]
TRACE No credentials in cache for URL http://packagecloud.localhost/khor/another-repo/pypi/simple/verysimplemodule/
TRACE Attempting unauthenticated request for http://packagecloud.localhost/khor/another-repo/pypi/simple/verysimplemodule/
TRACE checkout waiting for idle connection: ("http", packagecloud.localhost)
DEBUG starting new connection: http://packagecloud.localhost/
TRACE Http::connect; scheme=Some("http"), host=Some("packagecloud.localhost"), port=None
DEBUG connecting to 127.0.0.1:80
DEBUG connected to 127.0.0.1:80
TRACE http1 handshake complete, spawning background dispatcher task
TRACE waiting for connection to be ready
TRACE connection is ready
TRACE checkout dropped for ("http", packagecloud.localhost)
TRACE put; add idle connection for ("http", packagecloud.localhost)
DEBUG pooling idle connection for ("http", packagecloud.localhost)
TRACE Resource is modified because status is 200 and not 304
DEBUG Found modified response for: http://packagecloud.localhost/khor/another-repo/pypi/simple/verysimplemodule/
TRACE Cached request http://packagecloud.localhost/khor/another-repo/pypi/simple/verysimplemodule/ is storable because this is a shared cache and its response has a 'private' cache-control directive
TRACE Received package metadata for: verysimplemodule
TRACE Selecting candidate for verysimplemodule with range >=0.0.1, <0.0.1+ with 1 remote versions
DEBUG Searching for a compatible version of verysimplemodule (>=0.0.1, <0.0.1+)
TRACE Selecting candidate for verysimplemodule with range >=0.0.1, <0.0.1+ with 1 remote versions
TRACE Found candidate for package verysimplemodule with range >=0.0.1, <0.0.1+ after 1 steps: 0.0.1 version
TRACE Returning candidate for package verysimplemodule with range >=0.0.1, <0.0.1+ after 1 steps
TRACE Found candidate for package verysimplemodule with range >=0.0.1, <0.0.1+ after 1 steps: 0.0.1 version
TRACE Returning candidate for package verysimplemodule with range >=0.0.1, <0.0.1+ after 1 steps
DEBUG Selecting: verysimplemodule==0.0.1 [compatible] (verysimplemodule-0.0.1-py3-none-any.whl)
TRACE idle interval checking for expired
TRACE Cached request http://packagecloud.localhost/khor/another-repo/pypi/packages/verysimplemodule-0.0.1-py3-none-any.whl.metadata is storable because its response has a heuristically cacheable status code 200
TRACE Freshness lifetime heuristically assumed because of presence of last-modified header: 600s
DEBUG Found fresh response for: http://packagecloud.localhost/khor/another-repo/pypi/packages/verysimplemodule-0.0.1-py3-none-any.whl.metadata
TRACE Received built distribution metadata for: verysimplemodule==0.0.1
INFO add_decision: Id::<PubGrubPackage>(2) @ 0.0.1 without checking dependencies
TRACE Assigned packages: root==0a0.dev0, test==0.1.0, verysimplemodule==0.0.1
DEBUG Tried 2 versions: test 1, verysimplemodule 1
DEBUG all marker environments resolution took 0.198s
TRACE Resolution: ResolverEnvironment { kind: Universal { initial_forks: [], markers: true, include: {}, exclude: {} } }
TRACE Resolution edge: ROOT -> test
TRACE Resolution edge: 0a0.dev0 -> 0.1.0
TRACE Resolution edge: test -> verysimplemodule
TRACE Resolution edge: 0.1.0 -> 0.0.1
Resolved 2 packages in 204ms
TRACE pool closed, canceling idle interval
DEBUG Using request timeout of 30s
DEBUG Registry requirement already cached: verysimplemodule==0.0.1
TRACE Extracting file name=PackageName("verysimplemodule")
TRACE Extracted 4 files name=PackageName("verysimplemodule")
TRACE No entrypoints name=PackageName("verysimplemodule")
TRACE No data name=PackageName("verysimplemodule")
TRACE Writing installer metadata name=PackageName("verysimplemodule")
TRACE Writing record name=PackageName("verysimplemodule")
Installed 1 package in 1ms
+ verysimplemodule==0.0.1
DEBUG Released lock at `/test/.venv/.lock`
Thanks @nethsix. Unfortunately that doesn't reveal a lot 😅 -- looks like the cache hits only show up for the pre-redirect URLs and there's no logging of the artifact itself, just the .metadata variant.
I'm not sure what to make of that. Do the logs vary much when you clear uv's cache entirely?
(I also don't see any signs of authentication or 302 redirects at all in those logs -- could you say more about how you're authenticating to that index?)
it would be good to better understand the intended behavior here, since a signed URL strongly suggests that caching shouldn't be done
Pulling back from the tangent on URL components, I am a user and not operator of one of the Python Registries in question, but I believe that I can describe the design intent.
The Registry provides an index like one would expect.
The index has links not unlike the PyPI: ex https://packages.example.com/acme-corp/mirrors/pypi/packages/f1/13/63c0a02c44024ee16f664e0b36eefeb22d54e93531630bd99e237986f534/cowsay-6.1-py3-none-any.whl
(This url must be stable for among other things lockfiles to work)
The above URL redirects (302) to a signed url to S3 or some other object storage system. The HTTP cache headers align with the time for which the signed url is valid.
Activity
woodruffw commentedon May 19, 2025
Hi @mklein0, thanks for opening an issue.
I think the reason we don't cache this is because the various RFCs are ambiguous about temporary redirects vs. permanent ones: RFC 2616 doesn't proscribe caching for these ("cacheable" isn't qualified by SHOULD or similar), while RFC 7231 doesn't mention caching for 302 at all and RFC 7234 mentions only permanent redirects explicitly.
I'm not opposed to changing this per se, but I'd like to better understand why JFrog is using HTTP 302 instead of HTTP 301 for these redirects -- ideally this is something they'd fix on their own end, since other mirror indices don't appear to have this problem.
woodruffw commentedon May 19, 2025
(More broadly, I agree with the appraisal in pypa/pip#13396 (comment) -- it would be good to better understand the intended behavior here, since a signed URL strongly suggests that caching shouldn't be done.)
mklein0 commentedon May 19, 2025
As you pointed out the standards are vague on required behavior, so I interpretted it as a suggestion as well. I do not know how common the usage of a private CDN redirect is, so I thought to bring it up and see if there is a best practice.
I will take these two tickets to JFROG and have them advocate for a behavior or have them alter theirs. Thank you.
nethsix commentedon Jul 25, 2025
@woodruffw I believe JFrog is not using HTTP 301 for redirects likely the 'Location:' of the redirect has same url but different query component, i.e., the signature, e.g.,
X-Goog-Signatureis different each time?Not caching the initial request that results in the HTTP 302 redirect may not be that critical since the relevant content returned is tiny with just the
Location:being important.Would the concern be more of NOT caching the redirected
Location:url which is the larger piece of content being transferred. I did some tests, and is it correct to say that pip performs a cache hit/miss based on the entire url including the query components? This results in the caching NOT happening because each time theX-Goog-Signaturechanges?woodruffw commentedon Jul 25, 2025
Sorry, could you clarify what you mean here? "A same URL but different query component" is not the same URL, since the query is part of the URL. I'm guessing you mean "same path component," but I want to make sure I understand correctly.
I think the pip maintainers would better be able to answer what pip specifically does -- all I know is what the caching middleware does, which pip may or may not tweak, do a superset of, etc.
For CacheControl specifically, we (currently) key off of the query string exactly as it appears in the request via
_urlnormhere:cachecontrol/cachecontrol/controller.py
Lines 65 to 83 in 9af76f7
(Based on my read of the RFCs, this is correct enough, and ignoring the query URL would cause overly permissive cache hits.)
nethsix commentedon Jul 26, 2025
@woodruffw Sorry that I was not clear with what I meant by "A same URL but different query component". I have an AWS cloudfront/s3 pre-signed url on hand so I will use that as an example.
The 'main' url is
https://d3njxchdk64paq.cloudfront.net/development/2/7/pypi/packages/8.whland this does not change for a pre-signed url. Then there is the query components/url parameters, i.e., everything after?, e.g.,Expires,Signature, etc., which usually changes each time the 'main' url is pre-signed.Thank you for clarifying with:
(a) This quote "... the query is part of the URL"
(b) The code snippet showing (a)
I think this is the crux of the discussion (I hope I didn't misinterpret anyone. At the very least our users (Buildkite/Packagecloud) are seeing the same phenomenon), i.e., each pre-signed url (whether google storage, aws cloudfront/s3, etc.) if considered with query component/string, is always different even though the pre-signed url is pointing to the same object in the bucket, thus cache hit never happens. This is because the pre-signed url contains query component/string with
ExpiresandSignature(for aws) and similar counterparts for other cloud, that is always changing each time a pre-signed url is generated.uv, which is apipalternative on the other hand behavesas expected, i.e., performing cache hit/miss based on just the url without considering the query components/strings.If you can help point to any RFCs that indicates cache hit/miss should take the full url including query components/string, it will be super helpful to determine where we can fix this issue.
woodruffw commentedon Jul 26, 2025
See this part of RFC 7234: https://datatracker.ietf.org/doc/html/rfc7234#section-4
Specifically, a cache MUST NOT reuse a response if the "effective request URI" does not match. That is then defined here: https://datatracker.ietf.org/doc/html/rfc7230#section-5.5
And to my read, the requirement here is to include the query component in the "effective request URI" (since it's a semantically relevant part of the URI).
I don't know exactly how uv behaves in this regard, but if you're seeing cache hits on signed URLs that vary by query component then I suspect that something, somewhere, is violating cache-control semantics. That, or there's some special casing around these kinds of signed URLs, since they're uncacheable by design.
However, to take a step back: can you confirm that what you're seeing is related to the original issue here? Specifically, is your first URL performing a 301 redirect or a 302 redirect?
woodruffw commentedon Jul 26, 2025
(Another possibility is that uv is doing application level catching instead of HTTP response caching. That would sidestep the problem entirely, but I don't know if that's actually what's happening.)
woodruffw commentedon Jul 28, 2025
I looked into this, and I don't believe uv is doing application-level caching -- it has HTTP caching middleware that (like CacheControl) doesn't cache the HTTP 302 redirect. As far as I can tell uv's caching should also invalidate on URL query changes (there are some places in uv where the query component is removed, but FWICT these don't affect the cache key).
nethsix commentedon Jul 29, 2025
This is what
pipis doing when I didpip install verysimplemodule===0.0.1the second time (I deleted the package after first install but left the cache untouched)[A] pip request for simple index
[B] pip gets 302 redirect
[C] pip says 302 is NOT in (200, 203, 300, 301, 308)
[D] pip looks up the redirect
Location:header which points to AWS cloudfront pre-signed url (which hasExpiresquery parameter that is constantly changing each this the same AWS cloudfront url is pre-signed[E] pip found not cache hit for the AWS cloudfront pre-signed url as it considers all query parameters when searching for cache hit
[F] pip downloads the python package from AWS cloudfront pre-signed url
Likewise when using
uvto downloadverysimplemodule===0.0.1the second time (I deleted the package after first install but left the cache untouched):[A] uv request for simple index
[B] uv seems to have found the cached url for simple index, and requested revalidation
[C] uv got modified response for simple index
[D] uv retrieved the newer simple index
[E] uv conclude that there is a cache hit for the python package so no need for redownload
notatallshaw commentedon Jul 29, 2025
I wasn't aware uv cached HTTP responses at all. I know it caches built packages and other application specific artifacts.
Can you point to what you mean in the uv codebase, prior art of other applications implementing your intended behavior is useful to make informed choices.
woodruffw commentedon Jul 29, 2025
@nethsix Thanks for the detailed logs, that's very helpful. Could you re-run the uv one at
TRACE? That might reveal more of the redirects.@notatallshaw Yep, uv uses a subset of the cache-control semantics -- you can see those here (in part): https://github.com/astral-sh/uv/blob/00efde06b61756f0f305fcf67b12db71a29063d3/crates/uv-client/src/httpcache/mod.rs
notatallshaw commentedon Jul 29, 2025
Thanks for the info.
This is probably obvious to everyone in the discussion but cache control is far more of a general middleware than pip's specific use case, and using http cache as the primary mechanism for catching is probably not perfect for pip, but cache control shouldn't be changed for that purpose.
I'm going to privately ping the uv team in case they have any input on the question of query components.
woodruffw commentedon Jul 29, 2025
Yep, agreed -- I also think (barring more evidence) that what CacheControl and pip are doing here is probably more correct, since the observed states (302 redirects and changing queries) are straightforwardly not cacheable at the HTTP layer. But to your point, this is a discrepancy between what pip semantically wants to cache (application-level contents) and what HTTP caching actually enables (responses).
I've already raised it internally 😉
nethsix commentedon Jul 30, 2025
@woodruffw Thanks for recommending running
uvwithTRACE, i.e.,-vvv. Would love you hear your interpretation of it.woodruffw commentedon Jul 31, 2025
Thanks @nethsix. Unfortunately that doesn't reveal a lot 😅 -- looks like the cache hits only show up for the pre-redirect URLs and there's no logging of the artifact itself, just the
.metadatavariant.I'm not sure what to make of that. Do the logs vary much when you clear uv's cache entirely?
(I also don't see any signs of authentication or 302 redirects at all in those logs -- could you say more about how you're authenticating to that index?)
nethsix commentedon Jul 31, 2025
@woodruffw I've uninstalled 'verysimplemodule' and deleted the cache dir, and performed
uvinstall again.The order of urls
uvcalls:Logs:
cache temporary redirects with explicit caching directives
cburroughs commentedon Jan 6, 2026
Pulling back from the tangent on URL components, I am a user and not operator of one of the Python Registries in question, but I believe that I can describe the design intent.
https://packages.example.com/acme-corp/mirrors/pypi/packages/f1/13/63c0a02c44024ee16f664e0b36eefeb22d54e93531630bd99e237986f534/cowsay-6.1-py3-none-any.whl