How to force discovery after generating packages?

[shareefj]

Hi there,

I'm auto-generating Python packages as a build step inserted before build_py but am failing to find a way to get setuptools to rerun the discovery phase.

What I've done is define a new setuptools.Command and then in my target package I add the following line:

setuptools.command.build.build.sub_commands.insert(0, ('generate_grpc_py_protobufs', None))

When I pip install my package, I can see that my build step runs after the point at which the src tree is parsed for packages. I've tried calling self.distribution.set_defaults(force=True) but when discovery:_analyse_package_layout() is called, discovery:_explicitly_specified() returns true. This seems to be due to the first discovery step setting discovery.dist.packages. The comments here seemed to suggest that my force should work.

My directory structure has a placeholder src directory that look like this:

src
└── package_name
    └── __init__.py

and then following auto-generation:

src
└── package_name
    ├── __init__.py
    ├── _version.py
    ├── sub_package_1
    │   ├── __init__.py
    │   └── sub_package_1.py
    └── sub_package_2
        ├── __init__.py
        └── sub_package_2.py

Should I be able to do this or am I doing something silly?

[shareefj]

So to expand on this and provide some examples, I have an example repo with 3 branches:

https://github.com/idex-biometrics/protobuf-grpc-distutils-example

Branch: main
This branch uses src-layout and auto-generates Python packages into a single level of hierarchy, src/helloworld.

$ pip install git+https://github.com/idex-biometrics/protobuf-grpc-distutils-example
$ tree  -I __pycache__ venv/lib/python3.10/site-packages/helloworld/
venv/lib/python3.10/site-packages/helloworld/
├── __init__.py
├── _version.py
├── client.py
├── hello_pb2.py
├── hello_pb2_grpc.py
└── server.py

0 directories, 6 files

Branch: deep-src-layout
This branch also uses src-layout but has an extra level of hierarchy. When the resulting package is installed, the protobuf Python modules are not present.

$ pip install git+https://github.com/idex-biometrics/protobuf-grpc-distutils-example@deep-src-layout
$ tree  -I __pycache__ venv/lib/python3.10/site-packages/helloworld/
venv/lib/python3.10/site-packages/helloworld/
├── __init__.py
├── _version.py
├── client.py
└── server.py

0 directories, 4 files

Branch: flat-layout
As above but a flat-layout. Again the auto-generated files are not installed:

$ pip install git+https://github.com/idex-biometrics/protobuf-grpc-distutils-example@flat-layout
$ tree  -I __pycache__ venv/lib/python3.10/site-packages/helloworld/
venv/lib/python3.10/site-packages/helloworld/
├── __init__.py
├── _version.py
├── client.py
└── server.py

0 directories, 4 files

I've just updated my protobuf-grpc-distutils Command to also write a MANIFEST.in file that adds a graft {output_dir} line and this seems to fix the flat-layout issues but doesn't fix the src-layout.

Branch: flat-layout (with MANIFEST.in)

$ tree  -I __pycache__ venv/lib/python3.10/site-packages/helloworld/
venv/lib/python3.10/site-packages/helloworld/
├── __init__.py
├── _version.py
├── client.py
├── hello
│   ├── __init__.py
│   ├── hello_pb2.py
│   └── hello_pb2_grpc.py
└── server.py

1 directory, 7 files

So I can work around this by using a flat-layout but what is the correct way to include auto-generated Python packages in a wheel?