Improve dependencies management #22
Conversation
The `dev.txt` requirements file contains all needed dependencies, and those versions already account for base requirements constraint because it is generated by `pip-compile` with the `--constraint` option. That's why installing dependencies from the `dev.txt` file is enough."
Updated the handling of requirements constraints by transitioning from using `pip-compile` to directly utilizing the `--constraints` option with `pip`.
This approach enhances clarity and reduces redundancy by isolating dependencies.
As development requirements no longer include base requirements, both need to be specified during installation. Using `--constraints` is unnecessary; the requirements `.txt` files already account for it.
| @@ -1,2 +1,3 @@ | |||
| -r base.txt | |||
There was a problem hiding this comment.
Normally, you need to include base.in to properly layer the direct requirements. You can actually add -c base.txt inside that file, and have an additional -c dev.txt inside this one. Adding constraints from other envs might cause confusion, I think.
There was a problem hiding this comment.
I view base requirements as core necessities, with dev (and potentially linters, prod or others later on) addressing according "additional" needs. This approach enables me to set up dependencies like building blocks, depending on use case objectives. Dependency separation is clear and understandable for me. However, I would really like to know how to do it better.
There was a problem hiding this comment.
This layering seems fine. Except that sometimes these things don't have to be related (like some linters may not really need all of your app deps to be installed and could be entirely separate, depending on what is set up and how).
There was a problem hiding this comment.
It's just that you shouldn't generally entangle pip-tools' input files (with the direct dependency lists) and the output files (constraints/lockfiles). Don't make inputs of one env depend on the outputs of some other. This creates additional artificial restrictions for no good reason.
| @@ -13,7 +13,7 @@ This is my educational project within IT KPI Python mentorship program by @webkn | |||
|
|
|||
| 2. Install requirements: | |||
| ```console | |||
| $ pip install -r requirements/base.in -c requirements/base.txt | |||
| $ pip install -r requirements/base.txt -r requirements/dev.txt | |||
There was a problem hiding this comment.
The -r must use .in files, while -c should point to actual constraints. Not all of the constraints will be installable under other environments but if they aren't abused as requirements, they won't cause such problems.
There was a problem hiding this comment.
Why the -r must use .in files, while -c should point to actual constraints? I've been reading articles on management to explore various solution options and also looked into dependency management at the warehouse, which also installs from .txt files. I understand that there may be factors I haven't considered, so I would really like to learn more about them.
There was a problem hiding this comment.
It may depend on the context how they are used. I'd argue that Warehouse treats them rather simplistically.
But in general, constraints are additional limitations to how the dependency resolver should behave, while requirements are what the user said the project needs directly. Misusing them, you basically declare that your project relies on every transitive dependency and uses each directly, which is plain wrong.
There are a lot of misconceptions around this, unfortunately. And pip-tools' own docs are far from ideal, reinforcing wrong ideas.
Warehouse installs from the files with the entire tree resolved and passes --no-deps, effectively disabling the dependency resolution and telling pip to skip trying to figure out the entire tree. They also run pip check after that to verify the integrity of those things installed because otherwise you can force pip to install incompatible projects.
A more complicated problem would be visible in projects having multiple environments that are platform-dependent. When you use pip-compile under Python 3.9 on macOS, it'll likely produce some more projects in the constraint file output. It can also omit some dependencies present on other platforms. So when you attempt to install from it under Python 3.12 on GNU/Linux, you may find yourself in a situation where you demand pip to install a package which is incompatible with your environment or not installable even, exploding on pip install. It may also have to find that project that was omitted in that other env, and it wouldn't be pinned.
When you do pip install -r requirements.in, you tell pip to install projects you definitely need, and their deps. Adding -c requirements.txt may help pip find the exact concrete project dists faster and more consistently. But if it has extra entries, the won't have any effect as you don't request them to be installed unconditionally.
With multiple envs, you need multiple constraint files per each OS+arch+Python combo. But you also need to track your direct deps which you do separately. With that, you'll have pairs of two files (in+txt) for each env you identify within a project. The .in files would be layered separately and the combination of several input layers would contribute to what will end up in the unified constraint file for the target environment.
Some time ago I was trying to record the misconceptions people tend to have around lockfiles: jazzband/pip-tools#1326 (comment).
| @@ -17,7 +17,7 @@ jobs: | |||
| python-version: 3.11 | |||
|
|
|||
| - name: Install requirements | |||
| run: python -m pip install -r requirements/dev.in -c requirements/dev.txt | |||
| run: python -m pip install -r requirements/base.txt -r requirements/dev.txt | |||
There was a problem hiding this comment.
Per other comments, such implementation is unidiomatic. Use explicit -c and -r for the proper environment, without referencing other unrelated dependency groups.
This PR enhances dependencies management with the following updates:
pip --constraintsfor constraints handling, replacing the use ofpip-compile --constraints.These changes aim to improve clarity and reduce redundancy in the project's dependency management.