Add tool to help with "bootstraping" build #4389

abravalheri · 2024-05-23T17:47:08Z

Summary of changes

Now that wheel is vendored, setuptools can be bootstrapped in a relatively easy way,
without the need of python setup.py install/develop/....

This PR adds a small script that follows PEP 517 (or at least a stripped down version of it, super-specialised for setuptools) + general instructions to the "Development Guide".

Closes #2828
Closes #2532 ? (pip install -e . already works fine for devs that have access to pip, so the only missing piece was bootstrapping)

Pull Request Checklist

Changes have tests
News fragment added in newsfragments/.
(See documentation for details)

abravalheri · 2024-05-23T17:48:20Z

pyproject.toml

@@ -167,6 +167,7 @@ exclude = [
 	"newsfragments*",
 	"docs",
 	"docs.*",
+	"build*",


When testing localy, I found pesky build/lib/**/*.py files being included in the wheel if the build directory was not empty.

This is also the reason why I changed MANIFEST.in.

Ugh. That doesn't seem right... adding a concern that seemingly affects all builds for something that's only affected when a bootstrap was run. Perhaps instead the bootstrap routine could remove the build directory after running?

I think it would make more sense if the routine could remove the build directory before running...

I noted that if build exists find_packages will include it. Once it is listed in packages, changing MANIFEST.in does not seem to make an effect.

Would it make more sense if instead we do:

[tool.setuptools.packages.find] include = [ "setuptools*", "pkg_resources*", "_distutils_hack*", ] exclude = [ "*.tests", "*.tests.*", ]

I think this simplifies things a bit and it is optimised for the case when the probability of adding .py in places outside the intended package directories is more likely than adding new package directories.

If the probabilities are reversed, then it makes less sense.

Ugh. That doesn't seem right... adding a concern that seemingly affects all builds for something that's only affected when a bootstrap was run. Perhaps instead the bootstrap routine could remove the build directory after running?

Probably the reason why python -m build works is because build copies the code to a temporary directory? (I guess)
But that is not a PEP 517 requirement...

I created a reproducer for the observed problem in #4394.

(other than pip install . or pip wheel ., running the test suite with tox itself also produces the build directory, so if anyone tries to run tox before pip install ., they also end-up with spurious files).

abravalheri · 2024-05-23T17:54:16Z

pkg_resources/__init__.py

@@ -2680,7 +2680,7 @@ def parse_map(
            _data = data.items()
        else:
            _data = split_sections(data)
-        maps: Dict[str, Dict[str, "EntryPoint"]] = {}
+        maps: Dict[str, Dict[str, EntryPoint]] = {}


For some reason the linting tools started to complain.

Shouldn't we fix this in a separate PR or commit, since it's unrelated to the proposal?

Agreed, I will work on it (this PR was more of a proof of concept).

abravalheri · 2024-05-23T18:17:25Z

Hi @jaraco, do think we should pursue something like this for setuptools?

jaraco

LGTM. It's a little annoying that this change introduces the 'build' and 'dist' directory concerns. Maybe those can be dealt with separately.

I do wonder if Setuptools should be in the business of providing the bootstrapping tool. IMO, if a downstream packager can't use pip, they should write their own PEP 517 builder/installer (that is, they should maintain it and not setuptools).

@FFY00 How do you feel about the approach? Should Setuptools provide this or could it be maintained outside setuptools?

jaraco · 2024-05-24T10:39:03Z

pkg_resources/__init__.py

@@ -2680,7 +2680,7 @@ def parse_map(
            _data = data.items()
        else:
            _data = split_sections(data)
-        maps: Dict[str, Dict[str, "EntryPoint"]] = {}
+        maps: Dict[str, Dict[str, EntryPoint]] = {}


Shouldn't we fix this in a separate PR or commit, since it's unrelated to the proposal?

jaraco · 2024-05-24T10:44:24Z

pyproject.toml

@@ -167,6 +167,7 @@ exclude = [
 	"newsfragments*",
 	"docs",
 	"docs.*",
+	"build*",


Ugh. That doesn't seem right... adding a concern that seemingly affects all builds for something that's only affected when a bootstrap was run. Perhaps instead the bootstrap routine could remove the build directory after running?

jaraco · 2024-05-24T10:46:50Z

MANIFEST.in

+prune dist
+prune build


I'm surprised this behavior isn't the default. Is it the case that if a dist directory existed prior to this change, it would end up in the sdist? That seems like bad default behavior and would have come up in other projects. I'd like to avoid this concern if possible. It's another case where adopting setuptools_scm will also bypass the concern.

We probably don't need this one. The problem is likely to be with tool.setuptools.packages.find: #4394.

abravalheri

@jaraco, thank you very much for the review.

I suppose another alternative is to document something like the following:

$ git clone https://github.com/pypa/setuptools
$ cd setuptools
$ python3 -c 'from setuptools.build_meta import build_wheel; build_wheel("./dist")'
$ python3 -m zipfile -e ./dist/setuptools-*.whl $TARGET_DIR

(No custom script required)

The disadvantage of this approach is that building the wheel directly from the source tree may yield different results than the wheel that we upload to PyPI (building the wheel from the sdist is usually more consistent).

abravalheri · 2024-05-24T11:06:29Z

pkg_resources/__init__.py

@@ -2680,7 +2680,7 @@ def parse_map(
            _data = data.items()
        else:
            _data = split_sections(data)
-        maps: Dict[str, Dict[str, "EntryPoint"]] = {}
+        maps: Dict[str, Dict[str, EntryPoint]] = {}


Agreed, I will work on it (this PR was more of a proof of concept).

abravalheri · 2024-05-24T11:12:04Z

pyproject.toml

@@ -167,6 +167,7 @@ exclude = [
 	"newsfragments*",
 	"docs",
 	"docs.*",
+	"build*",


I think it would make more sense if the routine could remove the build directory before running...

I noted that if build exists find_packages will include it. Once it is listed in packages, changing MANIFEST.in does not seem to make an effect.

abravalheri · 2024-05-24T11:12:21Z

MANIFEST.in

+prune dist
+prune build


We probably don't need this one. The problem is likely to be with tool.setuptools.packages.find: #4394.

FFY00 · 2024-05-28T12:52:37Z

I do wonder if Setuptools should be in the business of providing the bootstrapping tool. IMO, if a downstream packager can't use pip, they should write their own PEP 517 builder/installer (that is, they should maintain it and not setuptools).

@FFY00 How do you feel about the approach? Should Setuptools provide this or could it be maintained outside setuptools?

I don't have any complains about this, but I'd note the reason why downstream packagers usually can't use pip is because it vendors dependencies, and devendoring them for bootstrap is a mess. In Arch Linux, for example, we would similarly want to devendor the wheel dependency.

That said, this script, however, would be helpful regardless.

nanonyme · 2024-07-28T09:10:59Z

I do wonder if Setuptools should be in the business of providing the bootstrapping tool. IMO, if a downstream packager can't use pip, they should write their own PEP 517 builder/installer (that is, they should maintain it and not setuptools).
@FFY00 How do you feel about the approach? Should Setuptools provide this or could it be maintained outside setuptools?

I don't have any complains about this, but I'd note the reason why downstream packagers usually can't use pip is because it vendors dependencies, and devendoring them for bootstrap is a mess. In Arch Linux, for example, we would similarly want to devendor the wheel dependency.

That said, this script, however, would be helpful regardless.

There's no reason to do this. There is a clear bootstrapping path starting from flit_core which provides you fully working pypa/build and pypa/installer which you can then use to process pypa/wheel and finally pypa/setuptools and pypa/pip. Setuptools is not the beginning of PEP517 bootstrap chain and does not need to implement special tools. It does, however, need to support building itself with in-tree backend which is currently broken with newest pyproject-hooks.

abravalheri commented May 23, 2024

View reviewed changes

abravalheri force-pushed the bootstrap-build branch from ffd519f to 23962c2 Compare May 23, 2024 18:16

abravalheri marked this pull request as ready for review May 23, 2024 18:51

abravalheri requested a review from jaraco May 23, 2024 19:37

jaraco approved these changes May 24, 2024

View reviewed changes

jaraco requested a review from FFY00 May 24, 2024 10:55

abravalheri commented May 24, 2024

View reviewed changes

abravalheri force-pushed the bootstrap-build branch from f41c856 to e8c733e Compare May 24, 2024 12:55

jaraco force-pushed the main branch from 809a9a7 to ffdf0bd Compare September 15, 2024 14:44

abravalheri added 12 commits November 11, 2024 16:52

Add bootstrap capabilities to setuptools using PEP 517 hooks

03738d4

Use subprocess to simplify tar and unzip

01a21a4

Further simplify bootstrap by delegating extraction to user

877f4e6

Make private CLI even more obvious

6be92e5

Add docs for bootstrapping

d0e3f51

Further caveats in bootstrap documentation

454c48f

Show default values in argparser

addc8dc

Avoid errors on Windows due to the lack of SYSTEMROOT env var

843abb7

Further simplify calling hooks

a04748a

Remove unneeded entries from MANIFEST.in

ca45091

Fix linting error

fae1c98

XFAIL on test files included in the wheel

62634d8

abravalheri force-pushed the bootstrap-build branch from e8740f7 to 62634d8 Compare November 11, 2024 17:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add tool to help with "bootstraping" build #4389

Add tool to help with "bootstraping" build #4389

abravalheri commented May 23, 2024 •

edited

Loading

abravalheri May 23, 2024 •

edited

Loading

jaraco May 24, 2024

abravalheri May 24, 2024

abravalheri May 24, 2024

abravalheri May 24, 2024 •

edited

Loading

abravalheri May 24, 2024

abravalheri May 23, 2024 •

edited

Loading

jaraco May 24, 2024

abravalheri May 24, 2024

abravalheri May 24, 2024

abravalheri commented May 23, 2024

jaraco left a comment

jaraco May 24, 2024

jaraco May 24, 2024

jaraco May 24, 2024

abravalheri May 24, 2024 •

edited

Loading

abravalheri left a comment •

edited

Loading

abravalheri May 24, 2024

abravalheri May 24, 2024

abravalheri May 24, 2024 •

edited

Loading

FFY00 commented May 28, 2024

nanonyme commented Jul 28, 2024

		prune dist
		prune build

		prune dist
		prune build

Add tool to help with "bootstraping" build #4389

Are you sure you want to change the base?

Add tool to help with "bootstraping" build #4389

Conversation

abravalheri commented May 23, 2024 • edited Loading

Summary of changes

Pull Request Checklist

abravalheri May 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abravalheri May 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abravalheri May 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abravalheri commented May 23, 2024

jaraco left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abravalheri May 24, 2024 • edited Loading

Choose a reason for hiding this comment

abravalheri left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abravalheri May 24, 2024 • edited Loading

Choose a reason for hiding this comment

FFY00 commented May 28, 2024

nanonyme commented Jul 28, 2024

abravalheri commented May 23, 2024 •

edited

Loading

abravalheri May 23, 2024 •

edited

Loading

abravalheri May 24, 2024 •

edited

Loading

abravalheri May 23, 2024 •

edited

Loading

abravalheri May 24, 2024 •

edited

Loading

abravalheri left a comment •

edited

Loading

abravalheri May 24, 2024 •

edited

Loading