Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve dependencies management #22

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Improve dependencies management #22

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
afc8105
Optimize requirements installation across CI jobs
Martolivna Feb 24, 2024
b0646d8
Switch to `pip --constraints` for constraints
Martolivna Feb 24, 2024
23dc13f
Exclude base requirements from dev requirements
Martolivna Feb 24, 2024
ac47ca9
Update README for requirements installation
Martolivna Feb 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ jobs:
python-version: 3.11

- name: Install requirements
run: python -m pip install -r requirements/dev.in -c requirements/dev.txt
run: python -m pip install -r requirements/base.txt -r requirements/dev.txt
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per other comments, such implementation is unidiomatic. Use explicit -c and -r for the proper environment, without referencing other unrelated dependency groups.


- name: Run tests
run: python -m manage test
Expand All @@ -34,7 +34,7 @@ jobs:
python-version: 3.11

- name: Install requirements
run: python -m pip install -r requirements/dev.in -c requirements/dev.txt
run: python -m pip install -r requirements/base.txt -r requirements/dev.txt

- name: Install pre-commit
run: python -m pip install --upgrade pre-commit
Expand Down
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ This is my educational project within IT KPI Python mentorship program by @webkn

2. Install requirements:
```console
$ pip install -r requirements/base.in -c requirements/base.txt
$ pip install -r requirements/base.txt -r requirements/dev.txt
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The -r must use .in files, while -c should point to actual constraints. Not all of the constraints will be installable under other environments but if they aren't abused as requirements, they won't cause such problems.

Copy link
Collaborator Author

@Martolivna Martolivna Mar 1, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why the -r must use .in files, while -c should point to actual constraints? I've been reading articles on management to explore various solution options and also looked into dependency management at the warehouse, which also installs from .txt files. I understand that there may be factors I haven't considered, so I would really like to learn more about them.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may depend on the context how they are used. I'd argue that Warehouse treats them rather simplistically.
But in general, constraints are additional limitations to how the dependency resolver should behave, while requirements are what the user said the project needs directly. Misusing them, you basically declare that your project relies on every transitive dependency and uses each directly, which is plain wrong.
There are a lot of misconceptions around this, unfortunately. And pip-tools' own docs are far from ideal, reinforcing wrong ideas.

Warehouse installs from the files with the entire tree resolved and passes --no-deps, effectively disabling the dependency resolution and telling pip to skip trying to figure out the entire tree. They also run pip check after that to verify the integrity of those things installed because otherwise you can force pip to install incompatible projects.

A more complicated problem would be visible in projects having multiple environments that are platform-dependent. When you use pip-compile under Python 3.9 on macOS, it'll likely produce some more projects in the constraint file output. It can also omit some dependencies present on other platforms. So when you attempt to install from it under Python 3.12 on GNU/Linux, you may find yourself in a situation where you demand pip to install a package which is incompatible with your environment or not installable even, exploding on pip install. It may also have to find that project that was omitted in that other env, and it wouldn't be pinned.

When you do pip install -r requirements.in, you tell pip to install projects you definitely need, and their deps. Adding -c requirements.txt may help pip find the exact concrete project dists faster and more consistently. But if it has extra entries, the won't have any effect as you don't request them to be installed unconditionally.

With multiple envs, you need multiple constraint files per each OS+arch+Python combo. But you also need to track your direct deps which you do separately. With that, you'll have pairs of two files (in+txt) for each env you identify within a project. The .in files would be layered separately and the combination of several input layers would contribute to what will end up in the unified constraint file for the target environment.

Some time ago I was trying to record the misconceptions people tend to have around lockfiles: jazzband/pip-tools#1326 (comment).

```


Expand Down
14 changes: 7 additions & 7 deletions requirements/base.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
#
# This file is autogenerated by pip-compile with Python 3.11
# This file is autogenerated by pip-compile with Python 3.12
# by the following command:
#
# pip-compile --constraint=dev.txt base.in
# pip-compile requirements/base.in
#
asgiref == 3.7.2
asgiref==3.7.2
# via django
django == 3.2.20
# via -r base.in
pytz == 2023.3
django==3.2.20
# via -r requirements/base.in
pytz==2023.3
# via django
sqlparse == 0.4.4
sqlparse==0.4.4
# via django
3 changes: 2 additions & 1 deletion requirements/dev.in
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
-r base.txt
-c base.txt

pylint
40 changes: 12 additions & 28 deletions requirements/dev.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,40 +1,24 @@
#
# This file is autogenerated by pip-compile with Python 3.11
# This file is autogenerated by pip-compile with Python 3.12
# by the following command:
#
# pip-compile --constraint=base.in dev.in
# pip-compile requirements/dev.in
#
asgiref == 3.7.2
# via
# -r base.txt
# django
astroid == 2.15.6
astroid==2.15.6
# via pylint
dill == 0.3.7
dill==0.3.7
# via pylint
django == 3.2.20
# via
# -c base.in
# -r base.txt
isort == 5.12.0
isort==5.12.0
# via pylint
lazy-object-proxy == 1.9.0
lazy-object-proxy==1.9.0
# via astroid
mccabe == 0.7.0
mccabe==0.7.0
# via pylint
platformdirs == 3.10.0
platformdirs==3.10.0
# via pylint
pylint == 2.17.5
# via -r dev.in
pytz == 2023.3
# via
# -r base.txt
# django
sqlparse == 0.4.4
# via
# -r base.txt
# django
tomlkit == 0.12.1
pylint==2.17.5
# via -r requirements/dev.in
tomlkit==0.12.1
# via pylint
wrapt == 1.15.0
wrapt==1.15.0
# via astroid
Loading