Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python images broke - can we get a changelog? #969

Open
black-snow opened this issue Sep 17, 2024 · 16 comments
Open

Python images broke - can we get a changelog? #969

black-snow opened this issue Sep 17, 2024 · 16 comments

Comments

@black-snow
Copy link

We recently upgraded our base images from, for example,

python:3.11-bookworm@sha256:f7543d9969bdc112dd9819ca642e14433fdacfe857f170f6b803392fc7e451ad

to

python:3.11-bookworm@sha256:157a371e60389919fe4a72dff71ce86eaa5234f59114c23b0b346d0d02c74d39

and that broke all our Python images. I'm trying to understand what exactly changed (and then how to fix it). I saw in the issues that setuptools and wheel were drop as well as something with ensurepip?

What I've noticed in the image itself so far is that the name of the python binary seems to have changed - my symlink to /usr/local/bin/python3.11 is dead, there's only /usr/local/bin/python.

@pataquets
Copy link

pataquets commented Sep 17, 2024

I'm getting build errors for an already-deployed, successfully building commitd, since yesterday.
Base image is pinned to :3.12-slim.
Googling the error mostly pop setuptools issues, but nothing about that on my Dockerfile, AFAIK (at least no change on that). So, @black-snow's guess looks reasonable to me (within my knowledge).

The error pops when building Django staticfiles, no code nor deps changed:

#12 [stage-0 7/9] RUN   DEFAULT_FROM_EMAIL=dummy   FEEDBACK_EMAIL=dummy   EMAIL_URL=consolemail://   DATABASE_URL=sqlite://:memory:   SECRET_KEY=dummy   STATICFILES_STORAGE=django.contrib.staticfiles.storage.StaticFilesStorage   python manage.py collectstatic --no-input --verbosity 2
#12 0.426 Traceback (most recent call last):
#12 0.426   File "/app/manage.py", line 15, in <module>
#12 0.426     execute_from_command_line(sys.argv)
#12 0.426   File "/usr/local/lib/python3.12/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
#12 0.427     utility.execute()
#12 0.427   File "/usr/local/lib/python3.12/site-packages/django/core/management/__init__.py", line 416, in execute
#12 0.427     django.setup()
#12 0.427   File "/usr/local/lib/python3.12/site-packages/django/__init__.py", line 24, in setup
#12 0.427     apps.populate(settings.INSTALLED_APPS)
#12 0.427   File "/usr/local/lib/python3.12/site-packages/django/apps/registry.py", line 91, in populate
#12 0.427     app_config = AppConfig.create(entry)
#12 0.427                  ^^^^^^^^^^^^^^^^^^^^^^^
#12 0.427   File "/usr/local/lib/python3.12/site-packages/django/apps/config.py", line 193, in create
#12 0.427     import_module(entry)
#12 0.427   File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
#12 0.427     return _bootstrap._gcd_import(name[level:], package, level)
#12 0.427            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#12 0.427   File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
#12 0.427   File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
#12 0.427   File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
#12 0.427   File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
#12 0.427   File "<frozen importlib._bootstrap_external>", line 995, in exec_module
#12 0.427   File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
#12 0.427   File "/usr/local/lib/python3.12/site-packages/rest_framework_simplejwt/__init__.py", line 1, in <module>
#12 0.427     from pkg_resources import DistributionNotFound, get_distribution
#12 0.427 ModuleNotFoundError: No module named 'pkg_resources'
#12 ERROR: process "/bin/sh -c DEFAULT_FROM_EMAIL=dummy   FEEDBACK_EMAIL=dummy   EMAIL_URL=consolemail://   DATABASE_URL=sqlite://:memory:   SECRET_KEY=dummy   STATICFILES_STORAGE=django.contrib.staticfiles.storage.StaticFilesStorage   python manage.py collectstatic --no-input --verbosity 2" did not complete successfully: exit code: 1
------

@pataquets
Copy link

pataquets commented Sep 17, 2024

In addition, same commit builds successfully with :3.9-slim.
Trying from :3.12-slim (which equals to :3.12.6-slim as of now) downwards, worked for me with :3.12.4-slim.
Fair enough.

@black-snow
Copy link
Author

Regarding my particular issue I'm not quite sure where it stems from as there's a bunch of moving pieces involved (python base image, minimal debian base, copying things around, poetry, venvs, ...) - but for now I'll just fix it with
RUN ln -s /usr/local/bin/python /usr/local/bin/python3.11.

I'm still curious about what exactly changed. Could we perhaps get a changelog we could check before bumping the digest? It's rather difficult to manually diff what's changed in between versions ;)

@LaurentGoderre
Copy link
Member

This looks very similar to #967

@tianon
Copy link
Member

tianon commented Sep 17, 2024

See also #957 (and other linked/related issues/PRs) 👀

@edmorley
Copy link
Contributor

edmorley commented Sep 17, 2024

Hi

These are two completely separate issues (in general it's best to open separate issues rather than re-using an existing open issue when the symptoms aren't the same):

@klausmyrseth
Copy link

klausmyrseth commented Sep 18, 2024

Setuptools was distributes with 3.12.1-4 but adruptedly stopped in 3.12.5 ... Is this something that permanent or is it just something that was done in the middle of a version chain by accident?

@edmorley
Copy link
Contributor

@klausmyrseth This was answered/linked to already above. Please read:

@klausmyrseth
Copy link

klausmyrseth commented Sep 18, 2024

ye but still quite interesting to do that on a minor verison update if you read my message it was a try to not be to harsh about the version usage and the fact that this is a breaking change for many. and to be completely sure as the update is HUGE (for many) and if someone suddenly brings it back that would break the builds again.

@edmorley
Copy link
Contributor

edmorley commented Sep 18, 2024

The reason I linked to that PR, is that your message asked if this change was done by accident - and the PR title and description make it clear that it wasn't by accident. It sounds that you are instead meaning "why was this change made deliberately" (which is a very reasonable question to ask, but it's still a different question), in which case there's more detail on the reasoning why in:
#964 (comment)

@black-snow
Copy link
Author

Thanks @edmorley !

Breaking changes suck and they are rather hard to avoid when the tag doesn't reflect a SemVer of the image but the underlying Python version (and who would want a 3.11-bookworm-1.17?).

So the remaining open question for me is if we could start providing a changelog. I pin all my digests and updates are deliberate, but it's veeeery cumbersome to find out what changed between two digests. I'll rename the issue to reflect that.

@black-snow black-snow changed the title Python 3.10, 3.11, 3.12 images broken for us Python images broke - can we get a changelog? Sep 18, 2024
@edmorley
Copy link
Contributor

edmorley commented Sep 18, 2024

I agree that normally a changelog is a must for a published project. However, I'm not sure a changelog would be very easy to implement in a useful way for this repository, due to it not being a typical type of package with versioned releases. How would you list the changes, when there isn't a version? By date? Also, the rate of commits on this repo is generally very low, so skim reading https://github.com/docker-library/python/commits/master/ should typically be sufficient to figure out exactly what changed in the last week or so - and any changes outside of version bumps are so rare the changelog will be redundant 99% of the time.

In general if you ever have a problem with any open source project, I'd recommend checking the GitHub releases section, followed by the commit history, then searching recently updated issues and PRs. In this case a search for the word "symlink" would have found the issue filed 5 days ago, describing exactly your issue:
https://github.com/docker-library/python/issues?q=is%3Aissue+symlink

@klausmyrseth
Copy link

What about bringing the links and names to the pull requests into a changelog that is easy accessible.

For me that would help me to see what's going on. For my sake I was very aware of the change python wise) but it can be hard to see what's going on with the docker. We had some code refering distutils in the old libs so I knew what to look for when the services refused to start suddently, but the first time I met the error it was hard to find the reason.

Also our thought process to remove the change when it was in v3.12.1 we thought you would not remove it until 13 as the version association brings that association to most users (like me)

@black-snow
Copy link
Author

@edmorley I disagree with a bunch of that ;)

Ofc I found similar issues, but do I want to rely on that someone else has already run into a similar problem and filed an issue for me to find before I bump the digest? Certainly not.

Do I want to identify all diff commits and read them in order to see if anything relevant might have changed? Nope.

I get that its somewhat difficult and many docker images are in a rather bad shape regarding this, yet writing a simple CHANGELOG.md when substantial things change seems easy enough to me.

@zikphil
Copy link

zikphil commented Sep 22, 2024

ye but still quite interesting to do that on a minor verison update if you read my message it was a try to not be to harsh about the version usage and the fact that this is a breaking change for many. and to be completely sure as the update is HUGE (for many) and if someone suddenly brings it back that would break the builds again.

Very interesting to do this in a minor version indeed. Just another reminder that we should never use third-party docker images on non-local environments

@black-snow
Copy link
Author

ye but still quite interesting to do that on a minor verison update if you read my message it was a try to not be to harsh about the version usage and the fact that this is a breaking change for many. and to be completely sure as the update is HUGE (for many) and if someone suddenly brings it back that would break the builds again.

Very interesting to do this in a minor version indeed. Just another reminder that we should never use third-party docker images on non-local environments

At least you shouldn't roll 'em out without proper testing and scanning.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants