-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
V2 support for Python 3.12 #8342
Comments
Thanks @ericbn - this was recently added to the v1 README. There is still more testing required for Python 3.12 on v2, but the v2 README will be updated when that is complete. Also for installing v2 of the AWS CLI we recommend using one of the options documented here: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html. |
@tim-finnigan not one of the options listed mention a package manager for the installation, which is a surprisingly common thing to want, so it's easy for a fellow user to see where @ericbn is coming from. |
Is there any time frame for adding python 3.12 support? Alpine Linux is upgrading python to 3.12. |
#8217 needs to be merged first. |
After updating dependencies, I got an issue when I was building the wheel:
After updating vendored six.py in vendored botocore like boto/botocore#2551, I can build the wheel, while the test suite fails in several places: test-failures.txt. At first glance, it seems more changes should be backported: |
@yan12125 I make two patches in PKGBUILD.
diff -Naur awscli-2.15.40/awscli/customizations/eks/kubeconfig.py awscli-2.15.40_/awscli/customizations/eks/kubeconfig.py
--- awscli-2.15.40/awscli/customizations/eks/kubeconfig.py 2024-04-20 11:06:32.658178400 +0800
+++ awscli-2.15.40_/awscli/customizations/eks/kubeconfig.py 2024-04-20 11:09:03.390341218 +0800
@@ -70,6 +70,13 @@
return name in [cluster['name']
for cluster in self.content['clusters'] if 'name' in cluster]
+ def __eq__(self, other):
+ return (
+ isinstance(other, Kubeconfig)
+ and self.path == other.path
+ and self.content == other.content
+ )
+
class KubeconfigValidator(object):
def __init__(self):
diff -Naur awscli-2.15.40/tests/functional/eks/test_kubeconfig.py awscli-2.15.40_/tests/functional/eks/test_kubeconfig.py
--- awscli-2.15.40/tests/functional/eks/test_kubeconfig.py 2024-04-20 11:06:32.568177140 +0800
+++ awscli-2.15.40_/tests/functional/eks/test_kubeconfig.py 2024-04-20 11:12:52.110419266 +0800
@@ -121,7 +121,7 @@
])
loaded_config = self._loader.load_kubeconfig(simple_path)
self.assertEqual(loaded_config.content, content)
- self._validator.validate_config.called_with(Kubeconfig(simple_path,
+ self._validator.validate_config.assert_called_with(Kubeconfig(simple_path,
content))
def test_load_noexist(self):
@@ -130,7 +130,7 @@
loaded_config = self._loader.load_kubeconfig(no_exist_path)
self.assertEqual(loaded_config.content,
_get_new_kubeconfig_content())
- self._validator.validate_config.called_with(
+ self._validator.validate_config.assert_called_with(
Kubeconfig(no_exist_path, _get_new_kubeconfig_content()))
def test_load_empty(self):
@@ -138,7 +138,7 @@
loaded_config = self._loader.load_kubeconfig(empty_path)
self.assertEqual(loaded_config.content,
_get_new_kubeconfig_content())
- self._validator.validate_config.called_with(
+ self._validator.validate_config.assert_called_with(
Kubeconfig(empty_path,
_get_new_kubeconfig_content()))
@@ -147,4 +147,4 @@
self.assertRaises(KubeconfigInaccessableError,
self._loader.load_kubeconfig,
current_directory)
- self._validator.validate_config.assert_not_called()
\ No newline at end of file
+ self._validator.validate_config.assert_not_called()
diff -Naur awscli-2.15.40/tests/unit/botocore/test_client.py awscli-2.15.40_/tests/unit/botocore/test_client.py
--- awscli-2.15.40/tests/unit/botocore/test_client.py 2024-04-20 15:00:39.465686191 +0800
+++ awscli-2.15.40_/tests/unit/botocore/test_client.py 2024-04-20 15:07:29.335115713 +0800
@@ -960,7 +960,7 @@
lines = [
(' Creates an iterator that will paginate through responses '
'from :py:meth:`MyService.Client.test_operation`.'),
- ' **Request Syntax** ',
+ ' **Request Syntax**',
' ::',
' response_iterator = paginator.paginate(',
" Foo='string',",
@@ -976,17 +976,17 @@
' :type Bar: string',
' :param Bar: Documents Bar',
' :type PaginationConfig: dict',
- ' :param PaginationConfig: ',
+ ' :param PaginationConfig:',
(' A dictionary that provides parameters to control '
'pagination.'),
- ' - **MaxItems** *(integer) --* ',
+ ' - **MaxItems** *(integer) --*',
(' The total number of items to return. If the total '
'number of items available is more than the value specified '
'in max-items then a ``NextToken`` will be provided in the '
'output that you can use to resume pagination.'),
- ' - **PageSize** *(integer) --* ',
+ ' - **PageSize** *(integer) --*',
' The size of each page.',
- ' - **StartingToken** *(string) --* ',
+ ' - **StartingToken** *(string) --*',
(' A token to specify where to start paginating. This is '
'the ``NextToken`` from a previous response.'),
' :returns: None',
diff -Naur awscli-2.15.40/tests/unit/botocore/test_waiters.py awscli-2.15.40_/tests/unit/botocore/test_waiters.py
--- awscli-2.15.40/tests/unit/botocore/test_waiters.py 2024-04-20 15:00:39.469019486 +0800
+++ awscli-2.15.40_/tests/unit/botocore/test_waiters.py 2024-04-20 15:04:08.810188250 +0800
@@ -648,7 +648,7 @@
(' Polls :py:meth:`MyService.Client.foo` every 1 '
'seconds until a successful state is reached. An error '
'is returned after 1 failed checks.'),
- ' **Request Syntax** ',
+ ' **Request Syntax**',
' ::',
' waiter.wait(',
" bar='string'",
diff -Naur awscli-2.15.40/tests/unit/botocore/test_utils.py awscli-2.15.40_/tests/unit/botocore/test_utils.py
--- awscli-2.15.40/tests/unit/botocore/test_utils.py 2024-04-20 12:09:38.883650919 +0800
+++ awscli-2.15.40_/tests/unit/botocore/test_utils.py 2024-04-20 12:11:56.434812142 +0800
@@ -1000,17 +1000,24 @@
'https://bucket.s3.amazonaws.com/key.txt')
-class TestSwitchToChunkedEncodingForNonSeekableObjects(unittest.TestCase):
- def test_switch_to_chunked_encodeing_for_stream_like_object(self):
- request = AWSRequest(
- method='POST', headers={},
- data=io.BufferedIOBase(b"some initial binary data"),
- url='https://foo.amazonaws.com/bucket/key.txt'
- )
- prepared_request = request.prepare()
- self.assertEqual(
- prepared_request.headers, {'Transfer-Encoding': 'chunked'}
- )
+def test_chunked_encoding_used_for_stream_like_object():
+ class BufferedStream(io.BufferedIOBase):
+ """Class to ensure seek/tell don't work, but read is implemented."""
+
+ def __init__(self, value):
+ self.value = io.BytesIO(value)
+
+ def read(self, size=-1):
+ return self.value.read(size)
+
+ request = AWSRequest(
+ method='POST',
+ headers={},
+ data=BufferedStream(b"some initial binary data"),
+ url='https://foo.amazonaws.com/bucket/key.txt',
+ )
+ prepared_request = request.prepare()
+ assert prepared_request.headers == {'Transfer-Encoding': 'chunked'}
class TestInstanceCache(unittest.TestCase): There are still a few test-errors only about |
Many thanks! Now I can build aws-cli-v2 for Python 3.12 on Arch Linux (not sure why there are other errors for you). I pushed your patches to the unofficial package https://aur.archlinux.org/packages/aws-cli-v2. A note about aws-cli-v2 package in Arch Linux: I was the maintainer for the official package extra/aws-cli-v2 until recently. I am unable to handle so many patches (ones for Python 3.12 & some others) while keeping the package up-to-date. Other Arch Linux Package Maintainers don't have enough time/energy, either. As a result, that package is dropped to AUR. |
FYI: alpine 3.20.0 removed aws-cli due to missing python 3.12 compatibility |
Sorry to spam, but is there any way the PRs mentioned by @onlined can be merged? Allowing aws-cli to run on Python 3.12 with such minimal changes would seem like it should be a high priority… |
@psychon The official way to run the aws-cli on Noble is to use the snap package. The deb package’s removal from Noble was influenced by the difficulty in keeping the package current via the Ubuntu SRU process. |
@marrek-az This might be a bit off-topic here, but okay: How do you know it is official? None of https://github.com/aws/aws-cli/tree/v2?tab=readme-ov-file#installation nor https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html mention it. I know that https://snapcraft.io/aws-cli and https://snapcraft.io/publisher/aws have green checkmarks indicating
At that point I had serious doubts about this snap not being a big supply chain attack. Edit: To clarify the previous sentence: Snapcraft claims that the snap is "official" and comes from AWS and AWS says they are not maintaining the snap. Thus, it seems to me that snapcraft is impersonating AWS. |
@psychon You ask some interesting questions, and I think there are some opportunities for improvement there. I can tell you that the snap is pre-installed on the official Ubuntu AMI on AWS, so you can check it from there. You can also diff the snap-installed version against the zip archive. It should be mentioned in the official Ubuntu documentation, and I’m going to ask them about doing so for consistency. Or do the PR for the doc update myself. One of those. As for the official aws-cli docs…, I’ll have to let someone else weigh in on that. |
@psychon Regarding the official status of the aws-cli snap on Ubuntu, a colleague pointed me to this over the weekend. It’s from the official Ubuntu documentation. EDIT: the link is now included in the official Ubuntu Noble 24.04 release notes. Hopefully, that will help. |
aws-cli got added to alpine 3.20 but it errors out :( |
@gattytto can you describe the error? best would be if you could open a issue on Alpine's issue tracker: <https://gitlab.alpinelinux.org/alpine/aports/-/issues>. Thanks!
|
right now alpine 3.19 works, but 3.20 docker.io/amd64/python:3.12.4-alpine3.20 aws cli gives this kind of error: / # pip install datetime
Traceback (most recent call last):
File "/usr/local/bin/pip", line 8, in <module>
sys.exit(main())
^^^^^^
File "/usr/local/lib/python3.12/site-packages/pip/_internal/cli/main.py", line 78, in main
command = create_command(cmd_name, isolated=("--isolated" in cmd_args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/pip/_internal/commands/__init__.py", line 114, in create_command
module = importlib.import_module(module_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 995, in exec_module
File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
File "/usr/local/lib/python3.12/site-packages/pip/_internal/commands/install.py", line 15, in <module>
from pip._internal.cli.req_command import (
File "/usr/local/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 19, in <module>
from pip._internal.index.package_finder import PackageFinder
File "/usr/local/lib/python3.12/site-packages/pip/_internal/index/package_finder.py", line 31, in <module>
from pip._internal.req import InstallRequirement
File "/usr/local/lib/python3.12/site-packages/pip/_internal/req/__init__.py", line 9, in <module>
from .req_install import InstallRequirement
File "/usr/local/lib/python3.12/site-packages/pip/_internal/req/req_install.py", line 40, in <module>
from pip._internal.operations.install.wheel import install_wheel
File "/usr/local/lib/python3.12/site-packages/pip/_internal/operations/install/wheel.py", line 40, in <module>
from pip._vendor.distlib.scripts import ScriptMaker
File "/usr/local/lib/python3.12/site-packages/pip/_vendor/distlib/scripts.py", line 16, in <module>
from .compat import sysconfig, detect_encoding, ZipFile
File "/usr/local/lib/python3.12/site-packages/pip/_vendor/distlib/compat.py", line 81, in <module>
import xmlrpc.client as xmlrpclib
File "/usr/local/lib/python3.12/xmlrpc/client.py", line 272, in <module>
if _try('%Y'): # Mac OS X
^^^^^^^^^^
File "/usr/local/lib/python3.12/xmlrpc/client.py", line 269, in _try
return _day0.strftime(fmt) == '0001'
^^^^^^^^^^^^^^^^^^^
AttributeError: 'datetime.datetime' object has no attribute 'times'. Did you mean: 'time'?``` |
@gattytto You error is entirely unrelated to this tool (you're not even running I see that you Python installation is in |
@WhyNotHugo thank you very much for your answer, here is a more specific example of how this does not work / # export AWS_BUCKET_NAME=ssd-bcc063a2c5d-1623-4875-a517-e5bfda543a6e AWS_ACCES
S_KEY_ID=DNKDQD4QETBWF1RIZU8C AWS_SECRET_ACCESS_KEY=q9wkHAw5RsQ5L658UN7a7dXiKsfi
z3KpueEfEmQl AWS_ENDPOINT_URL=http://rook-ceph-rgw-ssd-store.rook-ceph.svc
/ # aws s3 ls s3://$AWS_BUCKET_NAME
'datetime.datetime' object has no attribute 'times'
/ # FROM docker.io/amd64/python:3.12.4-alpine3.20
ENV CARGO_BUILD_JOBS=2
RUN pip install --upgrade pip
RUN pip install \
grpcio==1.64.1 \
boto3 \
solana \
base58 \
kubernetes \
requests \
debugpy \
grpcio-reflection==1.62.2 \
protobuf \
grpcio-health-checking==1.62.2 \
google-api-core[grpc] \
libcst==0.3.23 \
googleapis-common-protos>=1.63.2 \
proto-plus>=1.24.0 \
quart \
pynacl \
hypercorn
RUN apk add --no-cache \
gcc \
libffi-dev \
musl-dev \
openssl-dev \
aws-cli \
jq |
if I just change it to alpine3.19 for the base image it works just fine |
Could you please try it out to use official alpine images (e.g. `docker.io/library/alpine:latest`)? Also please try not to install python modules with pip, as this breaks system packages. Use system packages instead (apk add py3-*). Otherwise if it still fails, please discuss this issue in a more Alpine-related issue tracker, that this issue is only for Python 3.12 support on aws-cli.
|
@fossdd thank you very much for your answer, I am using an officially provided python image which is based in officially provided alpine image: FROM alpine:3.20 |
here's a more minimalistic test using recommended procedures as not adding anything by pip and installing aws-cli using apk. It still does not work FROM docker.io/amd64/python:3.12.4-alpine3.20
RUN apk add --no-cache \
aws-cli / # export AWS_BUCKET_NAME=ssd-bcc063a2c5d-1623-4875-a517-e5bfda543a6e AWS_ACCESS_KEY_ID=DNKDQD4QETBWF1RIZU8C AWS_SECRET_ACCESS_KEY=q9wkHAw5RsQ5L658UN7a7dXiKsfiz3KpueEfEmQl AWS_ENDPOINT_URL=http://rook-ceph-rgw-ssd-store.rook-ceph.svc
/ # aws s3 ls s3://$AWS_BUCKET_NAME
'datetime.datetime' object has no attribute 'times'
|
same happens if I Just use "FROM docker.io/python:3.12.4-alpine3.20" for the base image with no changes other than just apk add aws-cli |
@gattytto Test with the following Dockerfile instead:
On my end that works. However, note that alpine (specifically @fossdd who replied to you earlier) is performing their own patches to get this working (see #8689 (comment)). Those patches are not being supplied by AWS, so breakages occurring from the python:3.12.4-alpine3.20 image are some combination of issues with those patches and/or incompatibilities with libraries that are being updated by that python image. This issue is specifically a request for AWS to update aws-cli with official support for python 3.12 (not downstream patches from alpine). That's why @fossdd was directing you to report the issues on alpine-related trackers. |
thank you very much for the info and the answer @bobziuchovsky |
hopefully those aren't your real aws credentials here... |
It's a self hosted COSI implementation but yes lol |
Now that a version of V2 has been released that contains the fixes from #8917, does anything else need to be done to support Python 3.12? Or can this issue be closed? |
This issue is now closed. Comments on closed issues are hard for our team to see. |
Describe the feature
aws-cli V2 currently supports:
Add support for Python 3.12.x too.
Use Case
I installed aws-cli V2 using Homebrew, and it's the last tool I have installed still using Python 3.11. But more importantly, I believe this helps keep V2 up-to-date.
Proposed Solution
Add support for Python 3.12.x too.
Other Information
Python 3.12 was first released on 2023-10-02 and has EOL scheduled for 2028-10. See https://devguide.python.org/versions/
Acknowledgements
CLI version used
2.13.37
Environment details (OS name and version, etc.)
Darwin/21.6.0
The text was updated successfully, but these errors were encountered: