Failed to download distributions with unexpected BufError errors

[humanzz]

Hello,

My team develops locally mostly using MacOS, but we use a CI/CD (Amazon Linux 2) system as part of our development process.

We have a python package, with pyproject.toml and we use https://github.com/pyinvoke/invoke to manage our build scripts.

The main task in question now in tasks.py, uses uv pip install and looks as follows

UV_ENV = "UV_CACHE_DIR=<path> UV_INDEX_URL=<a non default index> "

@task
def develop(ctx):
    ctx.run(UV_ENV + "uv pip install --verbose -e '.[dev,testing]'", echo=True)

We use uv==0.1.5.

We're trying to move to using uv for its speed gains.
On MacOS, everything works as expected.
On the CI/CD host, we sometimes - more often than not - run into an error that originates from the uv pip install call in the develop task.

The error consistently looks as follows. The error sometimes changes the dependency it's failing to fetch.

error: Failed to download distributions
  Caused by: Failed to fetch wheel: numpy==1.26.4
  Caused by: Failed to extract source distribution
  Caused by: unexpected BufError

I'm attaching the --verbose output for a failed run and its successful retry. Note that I've slightly modified this to drop index urls, and package name, but the full list of dependencies should be there at the end of 2nd successful attempt.

uv_attempt1_fail.log
uv_attempt2_success.log

[zanieb]

Interesting thanks for the thorough report!

Related:

• include testing on amazonlinux:2023 #1534

This looks like an issue with our uv-extract crate. At the very least we should provide more information about what's gone wrong.

[humanzz]

Yeah. It's very tough to know what's going wrong here with such short message.

Another point on pip install and its output: verbose is too chatty, but omitting verbose is also too silent. pip on the other hand does print out some useful things like what's being downloaded, what's already satisfied, what index was used, etc. which are useful bits of info, that are much shorter than the chatty verbose output. It'd be nice if uv does a similar thing.

[zanieb]

Yep agree see #1569

[humanzz]

Awesome, I'll follow there for progress on that!

[humanzz]

I've started seeing a similar error on Mac (Apple Silicon), this time when using pip compile and uv==0.1.11 e.g.

UV_CACHE_DIR=<a cache dir> UV_INDEX_URL=<a custom index> uv pip compile pyproject.toml --all-extras --upgrade -o requirements.lock

error: Failed to download: ipython==8.22.1
  Caused by: The wheel ipython-8.22.1-py3-none-any.whl is not a valid zip file
  Caused by: an upstream reader returned an error: unexpected BufError
  Caused by: unexpected BufError

[zanieb]

It looks like maybe this comes from flate2 via async-compression via rs-async-zip which I think we call at

uv/crates/uv-extract/src/stream.rs

Line 22 in 8d72183

let mut zip = async_zip::base::read::stream::ZipFileReader::new(&mut reader);

For decompression this means that more input is needed to continue or the output buffer isn’t large enough to contain the result. The function can be called again after fixing both.

It seems like either:

• Your custom index is returning truncated output
• Our output buffer is overflowing somehow — I haven't looked into if this is even possible yet

[humanzz]

So, a bit more information - though likely a bit vague.
The setup I have, essentially runs a local proxy on my machine. I checked the logs of that proxy, to check entries when using pip and when using uv pip.

When using pip, logs look fine, no exceptions/errors.
When using uv pip, I did see some errors along the lines of

ERROR org.glassfish.jersey.server.ServerRuntime$Responder - An I/O error has occurred while writing a response message entity to the container output stream. - org.glassfish.jersey.server.internal.process.MappableException: java.io.IOException: Buffer overflow
...
...
Caused by: java.io.IOException: Buffer overflow.
	at org.glassfish.jersey.netty.connector.internal.JerseyChunkedInput.write(JerseyChunkedInput.java:211) ~[jersey-netty-connector-2.31.jar:?]
	at org.glassfish.jersey.netty.connector.internal.JerseyChunkedInput.write(JerseyChunkedInput.java:191) ~[jersey-netty-connector-2.31.jar:?]
	at org.glassfish.jersey.netty.connector.internal.JerseyChunkedInput.write(JerseyChunkedInput.java:182) ~[jersey-netty-connector-2.31.jar:?]
	at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:185) ~[jersey-common-2.35.jar:?]

I feel like I've read somewhere, on some of the issues, that uv, as an optimization, can stream files to read specific parts it might need. I wonder if maybe there's some weird interaction here, uv is reading the part and closing the connection, and that proxy is not properly handling that.

I'm not sure, really, but trying to hypothesize what might be wrong, given I don't have access to that proxy code, and I'm Rust-ignorant :(

[charliermarsh]

It's reasonable that we should just fall back to non-range requests when we see a BufError.

[zanieb]

I would recommend reporting that buffer overflow as a bug to the proxy you're using (if you can), it seems like the proxy is failing to buffer the streamed data correctly sometimes.

[chrisirhc]

In case it'd help someone else, I got this BufError error when the wheel file itself was corrupted.
In our case, the cache was serving a corrupted file (2.5MB), whereas the same file on public pypi is 9.6 MB, and uv was correctly failing when trying to use it.

It was just a bit difficult to understand what was the nature of the failure based on the error BufError.