cibuildwheel fails without diagnostic on dependency conflict

[tttapa]

Description

If there are dependency conflicts between cibuildwheel's requirement files and a project's build-system.requires dependencies, the build process simply fails without any meaningful diagnostics.

For example, Conan has a urllib3 requirement that conflicts with

cibuildwheel/cibuildwheel/resources/constraints-pyodide313.txt

Line 107 in 4b92d69

urllib3==2.5.0

When trying to add Conan as a build requirement in pyproject.toml, cibuildwheel just terminates without giving any indication about the actual cause of the error:

FROM python:3.13-bookworm
RUN pip install cibuildwheel==3.1.4
RUN mkdir foo
RUN cat <<EOF > foo/pyproject.toml
[project]
name="foo"
version="1.2.3"
[build-system]
requires = ["conan==2.19.1"]
build-backend = "foo"
EOF
RUN cibuildwheel foo --only cp313-pyodide_wasm32

[...]
Building wheel...

+ pyodide build /foo --outdir=/tmp/cibw-run-mj54e87y/cp313-pyodide_wasm32/built_wheel

Traceback (most recent call last):
  File "/tmp/cibw-run-mj54e87y/cp313-pyodide_wasm32/build/venv/lib/python3.13/site-packages/pyodide_build/pypabuild.py", line 366, in build
    built = _build_in_isolated_env(
        build_env, srcdir, str(outdir), "wheel", config_settings
    )
  File "/tmp/cibw-run-mj54e87y/cp313-pyodide_wasm32/build/venv/lib/python3.13/site-packages/pyodide_build/pypabuild.py", line 168, in _build_in_isolated_env
    install_reqs(build_env, env, builder.build_system_requires)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/cibw-run-mj54e87y/cp313-pyodide_wasm32/build/venv/lib/python3.13/site-packages/pyodide_build/pypabuild.py", line 138, in install_reqs
    env.install(
    ~~~~~~~~~~~^
        remove_avoided_requirements(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
        )
        ^
    )
    ^
  File "/tmp/cibw-run-mj54e87y/cp313-pyodide_wasm32/build/venv/lib/python3.13/site-packages/build/env.py", line 136, in install
    self._env_backend.install_requirements(requirements)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/tmp/cibw-run-mj54e87y/cp313-pyodide_wasm32/build/venv/lib/python3.13/site-packages/build/env.py", line 265, in install_requirements
    run_subprocess(cmd)
    ~~~~~~~~~~~~~~^^^^^
  File "/tmp/cibw-run-mj54e87y/cp313-pyodide_wasm32/build/venv/lib/python3.13/site-packages/build/_ctx.py", line 71, in run_subprocess
    subprocess.run(cmd, capture_output=True, check=True, env=env)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/cibw-run-mj54e87y/cp313-pyodide_wasm32/build/base/pbs/python/lib/python3.13/subprocess.py", line 577, in run
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/tmp/cibw-run-mj54e87y/cp313-pyodide_wasm32/build/venv/bin/python', '-m', 'pip', '--python', '/tmp/build-env-q1mdu4bs/bin/python', 'install', '--use-pep517', '--no-warn-script-location', '--no-compile', '-r', '/tmp/build-reqs-0nxveo_h.txt']' returned non-zero exit status 1.

ERROR Command '['/tmp/cibw-run-mj54e87y/cp313-pyodide_wasm32/build/venv/bin/python', '-m', 'pip', '--python', '/tmp/build-env-q1mdu4bs/bin/python', 'install', '--use-pep517', '--no-warn-script-location', '--no-compile', '-r', '/tmp/build-reqs-0nxveo_h.txt']' returned non-zero exit status 1.

                                                                       ✕ 3.62s
cibuildwheel: error: Command ['/tmp/cibw-run-mj54e87y/cp313-pyodide_wasm32/build/venv/bin/pyodide', 'build', '/foo', '--outdir=/tmp/cibw-run-mj54e87y/cp313-pyodide_wasm32/built_wheel'] failed with code 1. 

Preferably, this should print the pip output, along the lines of

ERROR: Cannot install -r cibw-req.txt (line 75), conan==2.19.1 and urllib3==2.5.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested urllib3==2.5.0
    requests 2.32.4 depends on urllib3<3 and >=1.21.1
    conan 2.19.1 depends on urllib3<2.1 and >=1.26.6

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

Build log

No response

CI config

No response

[joerick]

cibuildwheel is just passing through the output of pyodide build here, so perhaps this issue could be moved to https://github.com/pyodide/pyodide-build

[henryiii]

Looks like pyodide-build is not handling isolated builds correctly; these are supposed to be installed in a separate environment. It is an issue there, though, not here. Though I think we have most if not all of the pyodide maintainers here as well. ;)

[agriyakhetarpal]

Thanks for the report! We pass on the build to the pypa/build APIs that we wrap (_build_in_isolated_env() and _build_in_current_env()). I, too, don't understand why cibw-req.txt and the build-time dependencies are being mixed – that shouldn't really happen with an isolated build.

cc: @hoodmane @ryanking13 for visibility

[ryanking13]

Hmm, yeah, that's wierd. How is the contraints file (constraints-pyodide313.txt) passed during the build steps? It looks like pyodide-build is using that file inside the isolated build environment, when it should not.

[ryanking13]

cibuildwheel/cibuildwheel/platforms/pyodide.py

Lines 423 to 433 in 6331390

	if constraints_path:
	combine_constraints(build_env, constraints_path, identifier_tmp_dir)
	build_env["VIRTUALENV_PIP"] = pip_version
	call(
	"pyodide",
	"build",
	build_options.package_dir,
	f"--outdir={built_wheel_dir}",
	*extra_flags,
	env=build_env,
	)

The constraints file is passed to pyodide build command. I guess that constraints file should only be used when setting up python not when calling pyodide build?

[joerick]

It's a good question. On windows & macos, we do pass in the constraints like this. And it's good to be consistent between platforms.

But, linux doesn't have this option, and, the docs say that dependency-versions controls the versions of the tools that cibuildwheel uses, it has nothing to do with the versions of build-backend requirements. I wonder if we should change the behaviour across the board in fact, to not propagate dependency-versions into the environment of the build.

But I might be forgetting something... does build install something in the build venv that needs to be pinned? e.g. pip?

Even so, if we're not doing it on linux, I think there's a good argument to say we should try to be consistent across the board, and not pass this into the build.

Curious if you have a take on this @henryiii.

[henryiii]

I think this only affects Pyodide? I believe Windows and macOS do not set the constraints while the build is running, only when setting up.

[joerick]

Thats what I had thought too, but-

cibuildwheel/cibuildwheel/platforms/macos.py

Lines 466 to 490 in 03ce2f2

	if constraints_path:
	combine_constraints(
	build_env, constraints_path, identifier_tmp_dir if use_uv else None
	)
	match build_frontend.name:
	case "pip":
	# Path.resolve() is needed. Without it pip wheel may try to fetch package from pypi.org
	# see https://github.com/pypa/cibuildwheel/pull/369
	call(
	"python",
	"-m",
	"pip",
	"wheel",
	build_options.package_dir.resolve(),
	f"--wheel-dir={built_wheel_dir}",
	"--no-deps",
	*extra_flags,
	env=build_env,
	)
	case "build" | "build[uv]":
	if (
	use_uv
	and "--no-isolation" not in extra_flags
	and "-n" not in extra_flags

cibuildwheel/cibuildwheel/util/packaging.py

Lines 183 to 207 in 03ce2f2

	def combine_constraints(
	env: MutableMapping[str, str], /, constraints_path: Path, tmp_dir: Path | None
	) -> None:
	"""
	This will workaround a bug in pip<=21.1.1 or uv<=0.2.0 if a tmp_dir is given.
	If set to None, this will use the modern URI method.
	"""
	if tmp_dir:
	if " " in str(constraints_path):
	assert " " not in str(tmp_dir)
	tmp_file = tmp_dir / "constraints.txt"
	tmp_file.write_bytes(constraints_path.read_bytes())
	constraints_path = tmp_file
	our_constraints = str(constraints_path)
	else:
	our_constraints = (
	constraints_path.as_uri() if " " in str(constraints_path) else str(constraints_path)
	)
	user_constraints = env.get("PIP_CONSTRAINT")
	env["UV_CONSTRAINT"] = env["PIP_CONSTRAINT"] = " ".join(
	c for c in [our_constraints, user_constraints] if c
	)

our_constraints are added to PIP_CONSTRAINT which is applied to build_env.

[joerick]

I've verified this is the case on macOS with a local build. I set CIBW_DEPENDENCY_VERSIONS="packages: requests==2.32.0" and ran a build with a package that had requests in build-system.requires. When viewing the logs with verbose mode on, we see that build installs into the build env the version of requests that was specified in CIBW_DEPENDENCY_VERSIONS.

I think this is the wrong behaviour and we should change this behaviour across the board. I suspect that not many users are using CIBW_DEPENDENCY_VERSIONS to try to pin their package build requirements, so I think fixing it will be pretty low-impact.

(If anyone is wondering how to pin build-time requirements, you can use PIP_CONSTRAINT or UV_CONSTRAINT if you're building using uv.)