-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[v2] Distribute binaries for alpine / musl libc #4685
Comments
I tried to debug the issue, and couldn't figure out why it fails. It seems to me the install script is just a bit flaky. I tried to add the v2 label as well. |
The binaries we publish won't work on docker images based on Additionally, I don't think we'd ever release an alpine compatible binary like we're doing for our general linux binary. I think we'd be more inclined to just release a docker image that comes with our binary built on alpine. |
Thanks for your answer. Is it possible to document that here https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html under "Prerequisites for Linux", or make the error more user friendly in the alpine based image case? |
We'll make sure to get this updated in the user guide. I also like the idea of making a more user friendly error message for this case. |
@joguSD bummer, alpine is the standard for docker images. Especially with build / ci systems wanting to automate & not wanting to continually download very large docker images. Can use an official alpine docker image in a multi stage build to copy the binary into another docker image though. |
@chadgrant - can you give an example please of your multi-stage dockerfile? |
@BarakBD-Globality I do not have one but if the aws-alpine image ever gets created you can just do:
|
Too bad they don't support alpine. It is a standard image. |
@joguSD has there been any discussion or motion on this? It seems like a pretty big deal considering the prevalence of alpine. People like me use |
also can't install in |
Include information about awscli v2 not working in Alpine Linux as per aws/aws-cli#4685.
Interestingly if you use this as a base image the AWS cli will install and work properly. FROM adoptopenjdk/openjdk11:alpine I was able to do this using the following Dockerfile
|
The reason that the adoptopenjdk image works is because they include the missing glibc libraries. Alphine linux is based on 'musl glibc', a light-weight alternative to a fullblown glibc. The aws cli v2 binaries do not work with musl, they need a few more libraries to work. Instead of using the Java image, you can also use the Dockerfile below as an example how to run AWS CLI v2 on Alpine Linux:
The above will download the glibc libraries, download and install AWS CLI v2, and remove some stuff we probably don't need, such as auto-complete. The resulting image is about 100MB in size |
So what's the shortest path to get a I'd love to avoid to build my own |
@bentolor Since |
… tool Based on @blagerweij solution in the AWS CLI issue tracker: aws/aws-cli#4685 (comment)
thanks @blagerweij works really nicely |
Also thanks @blagerweij : I grasped the nettle and built a public, custom docker image by copy&pasting your solution: https://hub.docker.com/r/bentolor/docker-dind-awscli |
@bentolor I noticed in your repo that your Dockerfile uses |
@blagerweij See the image below from the official Docker image for illustration. The typical usage is:
Therefore: While it's true that the At least this is the usage pattern I'm aware of (i.e. on using Gitlab CI to build docker images). Maybe there are also direct |
One may use alpine image with glibc from https://hub.docker.com/r/frolvlad/alpine-glibc/ |
Tis is now failing on 2.7.31 and 2.7.30
Looking at a few places I use this, the last successful version/build is:
I have tried running through every versiuon and 2.7.28 and 2.7.29 give me a new error
Which makes 2.7.27 the last stable/working version. |
I encountered the same. Everything was fine up to 2.7.29. Things started to break since 2.7.30 and 2.7.31. Version 2.7.32 seems to have fixed the issue, although I'm still clueless on the botocore xform_name error message, and how the changes could affect this. PyInstaller has been upgraded to version 5.2, probably that's the root cause. But I can't find any obvious bugfix in 2.7.32, yet it seems to work. 🤷 |
aws-cli expects glibc shared objects which are not present on alpine. Alternatives considered: 1. Adding gcompat to the apk add list. The aws-cli still linked against missing shared objects. 1. Add missing shared objects manually like in aws/aws-cli#4685 (comment). This approached seems very brittle so I opted to use non-alpine images instead.
I found that this also works for:
But not with |
This thread is quite outdated: AWS CLI v2.10 has now proper support for Alpine / libmusl :
As you can see, we can use 'configure', 'make' and 'make install'. The above Dockerfile removes some 'nice to have' stuff such as autocomplete, documentation, etc. |
That's great! I was waiting for someone from AWS to update this thread appropriately. I haven't touched this for a while since I was still waiting, but let me try rebuilding. For cases like mine where we build and package things in CI, push them into a Linux repo, then have users install from the Linux repo — Does anyone know off the top of their head of there are IIRC, there was a time when this was not supported by this project. (But TBH, I might be thinking of something else.) |
The AWS CLI team recently published documentation for building CLI v2 from source here: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-source-install.html This contains a section "Alpine Linux container". |
Thanks for the update, it works nicely when building from source now without any gimmicks. Since the OP asked for pre-built distribution of alpine binaries, are there any future plans to get an official distribution in alpine? |
I just tested this. it works, however the image is 380MB. the docs says it's abt 150MB. 380MB is on par with the official amazonlinux2 based image (386MB) I only change version to latest (2.11.4) because the old link is broken. |
Please check the SO answer here: https://stackoverflow.com/a/61268529/1704634 Using Alpine 3.15 and Python 3.10 you can save a bit more (size is 108Mb). If you're really desperate to reduce the image size, you could consider to only keep the botocore data-files for the most commonly used services (ec2, s3, etc), since the botocore folder is about 70Mb. But I would recommend against doing that. But removing auto_completer and the examples helps a bit. Regarding the alpine package: Please see this commit: https://gitlab.alpinelinux.org/alpine/aports/-/commit/aa039cf358500ac471ba9f82529dba0c0fdc2887 Support is coming, although the Alpine devs decided to use the Python3 dependencies instead of using the portable-exe. The portable-exe is about 100Mb, using Python3 dependencies adds about 300Mb to your image. |
I also followed https://docs.aws.amazon.com/cli/latest/userguide/getting-started-source-install.html and it works for more recent versions.. FROM our.registry/alpine-python:3.10 AS builder
ARG INSTALL_VERSION=2.11.5
SHELL ["/bin/ash", "-eo", "pipefail", "-c"]
WORKDIR /target
# Grab install dependencies
RUN apk add --no-cache \
curl~=7.83 \
make~=4.3 \
cmake~=3.23 \
gcc~=11.2.1 \
g++~=11.2.1 \
libc-dev~=0.7.2 \
libffi-dev~=3.4.2 \
openssl-dev~=1.1.1 \
\
&& curl https://awscli.amazonaws.com/awscli-${INSTALL_VERSION}.tar.gz | tar -xz --strip-components 1 \
&& ./configure --with-download-deps --with-install-type=portable-exe \
&& make && make install \
\
&& rm -rf \
/usr/local/lib/aws-cli/aws_completer \
/usr/local/lib/aws-cli/awscli/data/ac.index \
/usr/local/lib/aws-cli/awscli/examples && \
find /usr/local/lib/aws-cli/awscli/data -name 'completions-1*.json' -delete && \
find /usr/local/lib/aws-cli/awscli/botocore/data -name 'examples-1.json' -delete This works fine. However, if I try So this has me a bit worried that this process is a bit fragile since we do rebuild historic images every now and again to patch OS level vulnerabilities while keeping the version the same. Anyone have any thoughts on where a perm base for the source for historic versions might be found? |
One thought is that support for Alpine / libmusl started happening with v2.10 going forward so I wouldn't think 2.9.x and lower would be supported or these tar.gz files would be available in the same manner as the new versions. |
From version v2.9.x they started to support Alpine/libmusl, however since v2.10 the build-system has been changed to use configure / make / make install, and use Python libraries (next to portable-exe). Regarding the source tar.gz files: you could consider to use the git tags: |
@vsimon - That makes sense. In-house, we're deprecating 2.9 in April so I guess we'll continue to build as we've been building w/ glibc in Alpine until then. Once we move off of 2.9, we'll switch our Dockerfiles to using configure/make/make install. What I didn't want was a separate build process for 2.9 and then everything else. @blagerweij - I did try that last week, but the source in github doesn't contain the configure/make/make install content that I'm interested in. It contains the Thanks for both of your comments. |
for those who are still having issues (like I was) and are having to do the installation procedure using
So, you just need to do: apk update
apk add aws-cli or if you want to specific a version like me, use: output: / # apk add "aws-cli=2.13.0-r0"
(1/62) Installing libbz2 (1.0.8-r5)
(2/62) Installing libexpat (2.5.0-r1)
(3/62) Installing libffi (3.4.4-r2)
(4/62) Installing gdbm (1.23-r1)
(5/62) Installing xz-libs (5.4.3-r0)
(6/62) Installing libgcc (12.2.1_git20220924-r10)
(7/62) Installing libstdc++ (12.2.1_git20220924-r10)
(8/62) Installing mpdecimal (2.5.1-r2)
(9/62) Installing ncurses-terminfo-base (6.4_p20230506-r0)
(10/62) Installing libncursesw (6.4_p20230506-r0)
(11/62) Installing libpanelw (6.4_p20230506-r0)
(12/62) Installing readline (8.2.1-r1)
(13/62) Installing sqlite-libs (3.41.2-r2)
(14/62) Installing python3 (3.11.4-r0)
(15/62) Installing python3-pycache-pyc0 (3.11.4-r0)
(16/62) Installing pyc (0.1-r0)
(17/62) Installing py3-certifi (2023.5.7-r0)
(18/62) Installing py3-certifi-pyc (2023.5.7-r0)
(19/62) Installing py3-cparser (2.21-r2)
(20/62) Installing py3-cparser-pyc (2.21-r2)
(21/62) Installing py3-cffi (1.15.1-r3)
(22/62) Installing py3-cffi-pyc (1.15.1-r3)
(23/62) Installing py3-cryptography (40.0.2-r1)
(24/62) Installing py3-cryptography-pyc (40.0.2-r1)
(25/62) Installing py3-six (1.16.0-r6)
(26/62) Installing py3-six-pyc (1.16.0-r6)
(27/62) Installing py3-dateutil (2.8.2-r3)
(28/62) Installing py3-dateutil-pyc (2.8.2-r3)
(29/62) Installing py3-distro (1.8.0-r2)
(30/62) Installing py3-distro-pyc (1.8.0-r2)
(31/62) Installing py3-colorama (0.4.6-r3)
(32/62) Installing py3-colorama-pyc (0.4.6-r3)
(33/62) Installing py3-docutils (0.19-r4)
(34/62) Installing py3-docutils-pyc (0.19-r4)
(35/62) Installing py3-jmespath (1.0.1-r1)
(36/62) Installing py3-jmespath-pyc (1.0.1-r1)
(37/62) Installing py3-urllib3 (1.26.15-r1)
(38/62) Installing py3-urllib3-pyc (1.26.15-r1)
(39/62) Installing py3-wcwidth (0.2.6-r2)
(40/62) Installing py3-wcwidth-pyc (0.2.6-r2)
(41/62) Installing py3-prompt_toolkit (3.0.38-r1)
(42/62) Installing py3-prompt_toolkit-pyc (3.0.38-r1)
(43/62) Installing py3-ruamel.yaml.clib (0.2.7-r1)
(44/62) Installing py3-ruamel.yaml (0.17.32-r0)
(45/62) Installing py3-ruamel.yaml-pyc (0.17.32-r0)
(46/62) Installing aws-cli-pyc (2.13.0-r0)
(47/62) Installing py3-awscrt-pyc (0.16.24-r0)
(48/62) Installing python3-pyc (3.11.4-r0)
(49/62) Installing aws-c-common (0.8.23-r0)
(50/62) Installing aws-c-cal (0.6.0-r0)
(51/62) Installing aws-c-compression (0.2.17-r0)
(52/62) Installing s2n-tls (1.3.45-r0)
(53/62) Installing aws-c-io (0.13.28-r0)
(54/62) Installing aws-c-http (0.7.11-r0)
(55/62) Installing aws-c-sdkutils (0.1.11-r0)
(56/62) Installing aws-c-auth (0.7.0-r0)
(57/62) Installing aws-checksums (0.1.16-r0)
(58/62) Installing aws-c-event-stream (0.3.1-r0)
(59/62) Installing aws-c-mqtt (0.8.14-r0)
(60/62) Installing aws-c-s3 (0.3.13-r0)
(61/62) Installing py3-awscrt (0.16.24-r0)
(62/62) Installing aws-cli (2.13.0-r0)
Executing busybox-1.36.1-r0.trigger
OK: 175 MiB in 77 packages
/ # aws --version
aws-cli/2.13.0 Python/3.11.4 Linux/5.15.49-linuxkit-pr source/x86_64.alpine.3 prompt/off
/ # which aws
/usr/bin/aws I hope I helped someone with this news 👍 |
I feel like this issue can be closed now. |
Thanks @jordanst3wart — closing this per comments above and will highlight this comment again for visibility:
|
|
Errors with:
The text was updated successfully, but these errors were encountered: