Failed installation : Illegal instruction (core dumped) on Linux ARM64

[dennis-glx]

Describe the bug

After following the instructions for installation on Linux ARM, I get the following error on trying to invoke the CLI tool.

$ aws --version
Illegal instruction (core dumped)

Expected Behavior

Expected to get the installed CLI version.

Current Behavior

$ aws --version
Illegal instruction (core dumped)

Some information of the system:

$ uname -r
Linux agx 4.9.253-tegra #1 SMP PREEMPT Mon Jul 26 12:19:28 PDT 2021 aarch64 aarch64 aarch64 GNU/Linux

Possible Solution

No response

Additional Information/Context

GDB Output

$  gdb /usr/local/aws-cli/v2/current/dist/aws

GNU gdb (Ubuntu 8.1.1-0ubuntu1) 8.1.1

Reading symbols from /usr/local/aws-cli/v2/current/dist/aws...(no debugging symbols found)...done.
(gdb) run
Starting program: /usr/local/aws-cli/v2/dist/aws
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
process 11080 is executing new program: /usr/local/aws-cli/v2/dist/aws
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".

Program received signal SIGILL, Illegal instruction.
OPENSSL_cpuid_setup () at /aws-crt-python/crt/aws-lc/crypto/fipsmodule/cpucap/cpu_aarch64_linux.c:132
132	/aws-crt-python/crt/aws-lc/crypto/fipsmodule/cpucap/cpu_aarch64_linux.c: No such file or directory.

CLI version used

Latest as of 28 June '23

Environment details (OS name and version, etc.)

Linux agx 4.9.253-tegra #1 SMP PREEMPT Mon Jul 26 12:19:28 PDT 2021 aarch64 aarch64 aarch64 GNU/Linux

[tim-finnigan]

Hi @dennis-glx thanks for reaching out. I think the Illegal instruction (core dumped) could indicate a compatibility issue with your architecture. Have you tried installing on another system? It might also be worth opening in an issue in the aws-crt-python repository given the output shared above.

[andrewhop]

What CPU are you attempting to install the CLI on, can you run lscpu and attach the output?

[andrewhop]

Based on the line number cpu_aarch64_linux.c:132 I believe you are hitting the same MIDR_EL1 issue that Valgrind was hitting. This was fixed in aws/aws-lc#1006 which was released in v1.10.0 of AWS-LC. The most recent version of the Python CRT includes aws-lc @ 80c3943 which is v1.11.0 which includes the fix. Can you try updating to a newer version of the AWS CLI?

[Dennis-Johnson]

Sure.

$ lscpu 

Architecture:                    aarch64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
CPU(s):                          8
On-line CPU(s) list:             0-3
Off-line CPU(s) list:            4-7
Thread(s) per core:              1
Core(s) per socket:              2
Socket(s):                       2
Vendor ID:                       Nvidia
Model:                           0
Model name:                      ARMv8 Processor rev 0 (v8l)
Stepping:                        0x0
CPU max MHz:                     2265.6001
CPU min MHz:                     115.2000
BogoMIPS:                        62.50
L1d cache:                       256 KiB
L1i cache:                       512 KiB
L2 cache:                        4 MiB
L3 cache:                        4 MiB
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Not affected
Vulnerability Spectre v1:        Mitigation; __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Branch predictor hardening
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fp asimd evtstrm aes pmull sha1 sha2 crc32 atom
                                 ics fphp asimdhp cpuid asimdrdm dcpop

[Dennis-Johnson]

If it helps, I ran the test code from the LC repo issue you mentioned.

#include <iostream>
#include <cstdint>
#include <asm/hwcap.h>
#include <sys/auxv.h>

static uint64_t armv8_cpuid_probe() {
  uint64_t val;
  __asm__ volatile("mrs %0, MIDR_EL1" : "=r" (val));
  return val;
}

int main() {
  uint32_t hwcap = getauxval(AT_HWCAP);
  std::cout << "hwcap = " << hwcap << "\n";
  std::cout << "HWCAP_CPUID = " << HWCAP_CPUID << "\n";
  uint32_t has_cpu_id = hwcap & HWCAP_CPUID;
  std::cout << "hwcap & HWCAP_CPUID = " << has_cpu_id << "\n";
  if (has_cpu_id) {
    std::cout << "armv8_cpuid_probe = " << armv8_cpuid_probe() << "\n";
  }
  return 0;
}

Output

hwcap = 73727
HWCAP_CPUID = 2048
hwcap & HWCAP_CPUID = 2048
armv8_cpuid_probe = 1309605952

[ArivCR7]

@andrewhop , I'm also getting the same issue when installing aws-cli v2 on arm64 processor. This is impacting our production. Can you please investigate and let us know the resolution.

[jhussainx]

It appears we may have hit this issue and managed to workaround it by installing an earlier version of awscli . Tested with version 2.0.30 , 2.3.0, 2.4.0, 2.8.0, 2.11.0 , 2.11.10, 2.11.12 and 2.11.13 which appear to be working .

Issue appears to occurs from release 2.11.14 on wards .

curl "https://awscli.amazonaws.com/awscli-exe-linux-aarch64-2.11.13.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

[tim-finnigan]

After further research by the team it appears that increasing the awscrt version ceiling should address this. Here is the PR for that: #8056. A related PR (aws/aws-lc#1006) was referenced in an earlier comment as well as awscrt v0.16.18.

[amberkushwaha]

after further research by the team it appears that the corrupted coding is now being settled by the clients end and also the mono-log was shifted to the server.it seems the issue is been settled now and if it reappears i will handle it by sanctioning it to the core team and log.

[amberkushwaha]

update awscrt requirements by the team it appears that the corrupt coding is now being settled by the clients end and also thr mono log was shifted to the servent it seems the issue is been settled now and if it reappears i will handle it by sanctioning it to the core team in it and remaining log modules.

[tim-finnigan]

The awscrt ceiling was raised to 0.19.12 in #8324. This should now be resolved in v2.13.35+ (CHANGELOG for reference.) If anyone is still seeing this or related issues please let us know. Thanks!

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.