Two packages but only one setupc.cfg

[buhtz]

I'm using the src folder layout and setup.cfg with a setup.py (just calling setup()).
Then I install my packages in Developer Mode via --editable. That works fine.

But I was wandering if it is possible to use one setup.cfg for two packages. The folder structure would look like this

project_folder
    tests
    src
        packageA
        packageB

When I do python3 -m pip --editable . I would assume that this installs two packages.

Of course I still tried and it doesn't work because you can not have two [metadata] sections in a setup.cfg.

Is there a way to handle that?

I want to have A and B in the same repo. There is no way to separate them.

[abravalheri]

@Codeberg-AsGithubAlternative-buhtz what is the error message you are seeing? (And could you share the relevant parts of your configuration file?)

In the Python packaging ecosystem the terminology is a bit abused, but for the sake of simplicity let's say that in one distribution (the stuff that you upload to PyPI) can have multiple importable packages. So theoretically there is nothing forbidding you to do that.

Of course it all depends on what you have added to your configuration files...

[buhtz]

Thank you for your quick reply. My apologize for that broad question. It wasn't clear for me what I really want. 😄

This is the repo structure with two folders in the src folder.

cliqt_demo
├── setup.cfg
├── setup.py
├── src
│   ├── bit
│   │   └── __init__.py
│   └── qt
│       └── __init__.py
└── tests

I know that config is not well but it gives an idea of my goal I hope:

$ cat setup.cfg
[metadata]
name = bit

[options]
packages =
    bit
    qt
package_dir =
    bit = src/bit
    qt = src/qt

Now I "install" it.

python3 -m pip install --editable .
Obtaining file:///home/user/ownCloud/my.work/cliqt_demo
Installing collected packages: bit
  Running setup.py develop for bit
Successfully installed bit

The output is not what I want. I would like to see something like this

Obtaining file:///home/user/ownCloud/my.work/cliqt_demo
Installing collected packages: bit
  Running setup.py develop for bit
Installing collected packages: qt
  Running setup.py develop for qt
Successfully installed bit, qt

OK but pip did installed something so I tried to do import what doesn't work either:

Python 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import bit
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'bit'
>>> import qt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'qt'
>>>

I don't want to hack something. If setuptools is not designed for something like this I will use another way. Maybe it was a stupid idea of an inexperienced python "developer".

For example I simply could create two totally separate repositories

cli_demo
├── setup.cfg
├── setup.py
├── src
│   ├── bit
│   │   └── __init__.py
└── tests

qt_demo
├── setup.cfg
├── setup.py
├── src
│   └── qt
│       └── __init__.py
└── tests

Or I could use subpackages:

cliqt_demo
├── setup.cfg
├── setup.py
├── src
│   └── bit
│       └── __init__.py
│       └── qt
│           └── __init__.py
└── tests

[abravalheri]

Thank you very much for sharing part of your config. I think I have an idea what is happening with the imports.

If you don't have a pyproject.toml file, pip will do an editable installation using an old method that does not support this particular piece of configuration:

package_dir =
    bit = src/bit
    qt = src/qt

The easiest way to deal with that is to replace it with the equivalent:

package_dir =
    = src  # Yes, the value starts with an `=` character

Alternatively you can also add a pyproject.toml file with the following:

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"
# or "setuptools.build_meta:__legacy__" if you need to import something
# from your local folders in your `setup.py` file.

If you do at least one of these changes you should be able to import both packages.

---

The output is not what I want. I would like to see something like this
Successfully installed bit, qt

The output of the pip command is not in the scope of the setuptools project, but what pip is doing here is showing the name you assigned to your distribution when you wrote the [metadata] name = bit in setup.cfg. I am afraid there is no way to achieve what you want by publishing a single .whl file.

Think about name as the name of your "distribution" instead of the name of the "importable packages". According to the PyPA spec each distribution will have a single name.

This name usually coincides with the name of a single "importable package" contained in the distribution, but this is not a hard rule. You could for example assign name = bit_and_qt, and in this case pip would show Successfully installed bit_and_qt.

[buhtz]

I think I understand the difference between a distribution (represented by a whl file) and a package inside that distribution.

As you described I would say choosing project.toml is the best way because it is the (future) standard. But I also would like to combine it with a setup.cfg because in my real projects I often use constructs like that
version = attr: buhtzology.__version__
This isn't possible with toml currently if I am correctly informed.

So I tried but still have problems. But maybe I didn't understood all of your advice's.

The project.toml

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

The setup.cfg

[metadata]
name = bit

[options]
packages =
    bit
    qt
package_dir =
    bit = src/bit
    qt = src/qt

The usual setup.py

from setuptools import setup
setup()

I can not import bit or import qt. I assume there is a missing link between the toml and the cfg file, right? In the setuptools docu I couldn't find a hint about that. There is just a toml-only or cfg-only solution.

Solution (?)

I played a bit around and poked in the dark

The setup.cfg

[metadata]
name = bit

The project.toml

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[tool.setuptools.package-dir]
package-dir = {
    "bit" = "src/bit",
    "qt" = "src/qt"
}

[buhtz]

Sorry I forgot a paragraph in my last section "Solution (?)" 😄
The section means yes it works after I poked around. So what I show in that section does work for me.

Thank you very much for helping out here.

[abravalheri]

I can not import bit or import qt. I assume there is a missing link between the toml and the cfg file, right? In the setuptools docu I couldn't find a hint about that. There is just a toml-only or cfg-only solution.

You can use both at the same time. To be honest, with the configuration you are showing, things should be working. I can demonstrate that by running the following commands:

rm -rf /tmp/cliqt_demo
mkdir -p /tmp/cliqt_demo/src/bit
mkdir -p /tmp/cliqt_demo/src/qt

touch /tmp/cliqt_demo/src/bit/__init__.py
touch /tmp/cliqt_demo/src/qt/__init__.py

cd /tmp/cliqt_demo

cat <<EOF > setup.cfg
[metadata]
name = bit

[options]
packages =
    bit
    qt
package_dir =
    bit = src/bit
    qt = src/qt
EOF

cat <<EOF > setup.py
from setuptools import setup
setup()
EOF

cat <<EOF > pyproject.toml
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"
EOF

python3 -m venv .venv
.venv/bin/python -m pip install -U pip
.venv/bin/python -m pip install -e .
.venv/bin/python -c 'import bit, qt; print(bit, qt)'
# Will output:
# <module 'bit' from '/tmp/cliqt_demo/src/bit/__init__.py'> <module 'qt' from '/tmp/cliqt_demo/src/qt/__init__.py'>

Maybe something is getting cached?
Can you try completely uninstalling the bit package in your environment, removing the temporary build artifacts (e.g. rm -rf build dist *.egg-info) and trying again?

---

I often use constructs like that version = attr: buhtzology.__version__
This isn't possible with toml currently if I am correctly informed.

Some things fields can be dynamically derived via pyproject.toml as well, they are listed in https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html#dynamic-metadata. The procedure is slightly different though, first you need to say which fields are dynamic and then you tell setuptools how to find them. The example in the docs should give you a better idea.

NOTE 1: Please be aware that "raw" setuptools configuration only allows the attr: directive for the version field, even in setup.cfg: https://setuptools.pypa.io/en/latest/userguide/declarative_config.html#metadata

NOTE 2: If you are doing configuration with setup.cfg or pyproject.toml in a project that uses the src-layout, you should be able to safely omit the package_dir and packages configuration fields (setuptools should be able to find those things out automatically).

[buhtz]

I opened a separate but related question (#3725) about using pyproject.toml only.