Support multiple openssl versions #5996

th0ma7 · 2024-01-28T15:20:20Z

Description

Re-integrate support for armv5 using openssl.

Fixes #

Checklist

Build rule all-supported completed successfully --> from within cross/openssl & cross/openssl-latest
New installation of package completed successfully
Package upgrade completed successfully (Manually install the package again)
Package functionality was tested
Any needed documentation is updated/created

Type of change

How?

openssl/
├── openssl-1.1 (if armv5 or old-ppc)
├── openssl-3.1 (else)
└── openssl-latest (currently unused - v3.2.0)

IMPORTANT

This in turn will enable by default openssl3 for all other packages which hasn't yet been migrated. On later updates some may fail and require to enforce using openssl-1.1.

th0ma7 · 2024-01-28T15:24:22Z

Tested build:

lrwxrwxrwx 1 spksrc spksrc     13 Jan 28 15:07 work-88f6281-6.2.4/install/usr/local/openssl-main/lib/libssl.so -> libssl.so.1.1
-rwxr-xr-x 1 spksrc spksrc 780355 Jan 28 15:07 work-88f6281-6.2.4/install/usr/local/openssl-main/lib/libssl.so.1.1
lrwxrwxrwx 1 spksrc spksrc     11 Jan 28 15:08 work-aarch64-6.2.4/install/usr/local/openssl-main/lib/libssl.so -> libssl.so.3
-rwxr-xr-x 1 spksrc spksrc 858080 Jan 28 15:08 work-aarch64-6.2.4/install/usr/local/openssl-main/lib/libssl.so.3
lrwxrwxrwx 1 spksrc spksrc     11 Jan 28 15:08 work-aarch64-7.1/install/usr/local/openssl-main/lib/libssl.so -> libssl.so.3
-rwxr-xr-x 1 spksrc spksrc 949240 Jan 28 15:08 work-aarch64-7.1/install/usr/local/openssl-main/lib/libssl.so.3
lrwxrwxrwx 1 spksrc spksrc     11 Jan 28 15:08 work-armv7-6.2.4/install/usr/local/openssl-main/lib/libssl.so -> libssl.so.3
-rwxr-xr-x 1 spksrc spksrc 840532 Jan 28 15:08 work-armv7-6.2.4/install/usr/local/openssl-main/lib/libssl.so.3
lrwxrwxrwx 1 spksrc spksrc     11 Jan 28 15:09 work-armv7-7.1/install/usr/local/openssl-main/lib/libssl.so -> libssl.so.3
-rwxr-xr-x 1 spksrc spksrc 868980 Jan 28 15:09 work-armv7-7.1/install/usr/local/openssl-main/lib/libssl.so.3
lrwxrwxrwx 1 spksrc spksrc     11 Jan 28 15:10 work-comcerto2k-7.1/install/usr/local/openssl-main/lib/libssl.so -> libssl.so.3
-rwxr-xr-x 1 spksrc spksrc 680388 Jan 28 15:10 work-comcerto2k-7.1/install/usr/local/openssl-main/lib/libssl.so.3
lrwxrwxrwx 1 spksrc spksrc     11 Jan 28 15:10 work-evansport-6.2.4/install/usr/local/openssl-main/lib/libssl.so -> libssl.so.3
-rwxr-xr-x 1 spksrc spksrc 792868 Jan 28 15:10 work-evansport-6.2.4/install/usr/local/openssl-main/lib/libssl.so.3
lrwxrwxrwx 1 spksrc spksrc     11 Jan 28 15:11 work-evansport-7.1/install/usr/local/openssl-main/lib/libssl.so -> libssl.so.3
-rwxr-xr-x 1 spksrc spksrc 833468 Jan 28 15:10 work-evansport-7.1/install/usr/local/openssl-main/lib/libssl.so.3
lrwxrwxrwx 1 spksrc spksrc     11 Jan 28 15:11 work-hi3535-6.2.4/install/usr/local/openssl-main/lib/libssl.so -> libssl.so.3
-rwxr-xr-x 1 spksrc spksrc 847057 Jan 28 15:11 work-hi3535-6.2.4/install/usr/local/openssl-main/lib/libssl.so.3
lrwxrwxrwx 1 spksrc spksrc     11 Jan 28 15:12 work-qoriq-6.2.4/install/usr/local/openssl-main/lib/libssl.so -> libssl.so.3
-rwxr-xr-x 1 spksrc spksrc 853964 Jan 28 15:12 work-qoriq-6.2.4/install/usr/local/openssl-main/lib/libssl.so.3
lrwxrwxrwx 1 spksrc spksrc     11 Jan 28 15:12 work-x64-6.2.4/install/usr/local/openssl-main/lib/libssl.so -> libssl.so.3
-rwxr-xr-x 1 spksrc spksrc 929264 Jan 28 15:12 work-x64-6.2.4/install/usr/local/openssl-main/lib/libssl.so.3
lrwxrwxrwx 1 spksrc spksrc     11 Jan 28 15:12 work-x64-7.1/install/usr/local/openssl-main/lib/libssl.so -> libssl.so.3
-rwxr-xr-x 1 spksrc spksrc 964960 Jan 28 15:12 work-x64-7.1/install/usr/local/openssl-main/lib/libssl.so.3

NOTE: Latest is not being used currently but builds ok.

th0ma7 · 2024-01-28T15:34:17Z

@hgy59 ready for a review.

This in preparation to republishing python-3.10 and updating python-3.11 with re-enablement of both armv5 and qoriq.

hgy59 · 2024-01-28T18:10:31Z

@th0ma7 please read OpenSSL release strategy

Version 3.2 will be supported until 2025-11-23
Version 3.1 will be supported until 2025-03-14
Version 3.0 will be supported until 2026-09-07 (LTS).
Versions 1.1.1 and 1.0.2 are no longer supported. Extended support for 1.1.1 and 1.0.2 to gain access to security fixes for those versions is available.
Versions 1.1.0, 1.0.1, 1.0.0 and 0.9.8 are no longer supported.

OpenSSL 1 is EOL (1.1.1 since 2023-09-23)

The only LTS Version of OpenSSL is 3.0. I noticed this after releasing cross/openssl3 with 3.1.x and was hoping that 3.2 will be an LTS version too. But this is not the case.

So the best solution for the OpenSSL dependency would be to move towards OpenSSL 3.0.x and to remove OpenSSL 1.1.1 as soon as possible.

BTW:
since cross/openssl3 successfully builds for OLD_PPC_ARCHS and ARMv5_ARCHS we don't need this kind of separation of openssl version.
IMHO this looks like an issue with cryptography only and we should not downgrade openssl for other packages.

th0ma7 · 2024-01-28T18:18:49Z

@hgy59 there is a few points in here:

agreed, we can certainly downgrade to v3.0, and keep openssl-latest to ... well latest for future reference.
as for v1.1.1 the issue is twofold, indeed it's EOL but on the other hand for instance with python it won't work on armv5 due to missing libatomic (needed in conjunction with a "functional" openssl).

since cross/openssl3 successfully builds for OLD_PPC_ARCHS and ARMv5_ARCHS we don't need this kind of separation of openssl version.

Then this needs to be retested against python. Last I played with that python wasn't building ok due to atomic. If that's no longer the case then agreed, lets remove it entirely. If it doesn't work, then question is: do we decommission armv5 as now unsupported?

IMHO this looks like an issue with cryptography only and we should not downgrade openssl for other packages.

Currently armv5 is being "decommissionned" from multiple builds due to the openssl/libatomic issue. Some more testing is then needed to confirm what and when this breaks.

EDIT: Maybe the issue isn't existing with version 3.0.x ?

hgy59 · 2024-01-28T18:25:52Z

If we cannot solve the libatomic issue, we finally have to say goodbye to Python 3.11 for ARMv5 archs.

Or - if you really want it - build python 3.11 for ARMv5 with openssl 1.1.1 but do not affect other packages that use openssl3 for ARMv5 archs.

hgy59 · 2024-01-28T18:26:31Z

EDIT: Maybe the issue isn't existing with version 3.0.x ?

let's see 🤞

hgy59 · 2024-01-28T20:33:35Z

digging into this:

We build cross/openssl3 for ARMv5 and OLD_PPC with the configure argument no-threads to avoid dependency of libatomic.

This is required for both OpenSSL versions 3.1.4 and 3.0.12, i.e. downgrade to openssl 3.0 will not solve this issue.

Since the build of openssl3 succeeds for all archs, I guess cross/cryptography does not regard that openssl is built without the need of libatomic, or cryptography has its own dependency for atomics.

th0ma7 · 2024-01-28T21:15:53Z

So this is what's going on when building with a no-thread or no libatomic with python310:

checking for stdatomic.h... no
checking for builtin __atomic_load_n and __atomic_store_n functions... no
checking for ensurepip... no
checking if the dirent structure of a d_type field... yes
checking for the Linux getrandom() syscall... no
checking for the getrandom() function... no
checking for library containing shm_open... none required
checking for shm_open... yes
checking for shm_unlink... yes
checking for arm-marvell-linux-gnueabi-pkg-config... /usr/bin/pkg-config
checking whether compiling and linking against OpenSSL works... yes
checking for --with-openssl-rpath...
checking whether OpenSSL provides required ssl module APIs... yes
checking whether OpenSSL provides required hashlib module APIs... yes
checking for --with-ssl-default-suites... openssl
checking for --with-builtin-hashlib-hashes... md5,sha1,sha256,sha512,sha3,blake2

....

The necessary bits to build these optional modules were not found:
_tkinter                                                       
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


Failed to build these modules:
_hashlib              _ssl                                     


Could not build the ssl module!
Python requires a OpenSSL 1.1.1 or newer
Custom linker flags may require --with-openssl-rpath=auto

In turn this disable ssl from python altogether ... leading to the download errors seen with pip at installation time.

EDIT: I also tried the --with-openssl-rpath=auto but no luck.

hgy59 · 2024-01-28T21:24:19Z

So this is what's going on when building with a no-thread or no libatomic with python310:

So how was it possible to build python310 v3.10.13-19 for ARMv5?

th0ma7 · 2024-01-28T21:41:43Z

It should have read python-311 ... but checking with my python-3.10 test build and issue is there as well. I believe it actually started hapening during the 3.10.x cycle and had raise an issue about it or followed one (that I can't find anymore).

So issue is (unless something was overlooked):

we build armv5 with openssl-1.x (at least for python)
we drop armv5

th0ma7 · 2024-01-28T21:45:37Z

So how was it possible to build python310 v3.10.13-19 for ARMv5?

@hgy59 re-reading ... it is possible, but will lack _ssl thus unable to download anything at install time with pip. That's the issue.

hgy59 · 2024-01-28T22:25:03Z

So how was it possible to build python310 v3.10.13-19 for ARMv5?

@hgy59 re-reading ... it is possible, but will lack _ssl thus unable to download anything at install time with pip. That's the issue.

ok, got it:

/spksrc/spk/python310/work-88f6281-6.2.4/Python-3.10.12/Modules/_ssl.c:67:4: error: #error "OPENSSL_THREADS is not defined, Python requires thread-safe OpenSSL"

th0ma7 · 2024-01-28T23:01:52Z

Exactly!

I'd be willing to declare armv5 best effort with a banner on our website that explains the risks? Actually dsm6 altogether is in that boat.

Also, we may be able to use red hat, Ubuntu or other LTS distribution who may provide additional patches that can be easily applied where we maintain our own backport

hgy59 · 2024-02-02T15:46:27Z

@th0ma7 ARMv5 toolchains come with gcc 4.6.4 and glibc 2.15 but not with libatomic.

Even that gcc supports atomic operations since gcc 4.4, the toolchain does not include libatomic (but has libpthread).
https://gcc.gnu.org/projects/cxx-status.html#cxx11

Some information on libatomic is availabale at https://gcc.gnu.org/wiki/Atomic

If we could create newer gcc for ARMv5 archs (i.e. gcc 4.9.3 with glibc 2.20) we might get libatomic as part of gcc.

Otherwise we must drop openssl 3 and python 3.11 support for ARMv5.

th0ma7 · 2024-02-03T18:36:47Z

I don't think that would work as further reading in the links you provided it also requires a 3.1 or newer kernel (away from keyboard but if i recall armv5 runs a 2.6 kernel)

If that's right then options are:

Dropping arnv5 entirely
Using openssl 1.1 default with ⚠️ to community
Drop python support for armv5

I'd personally go with 2 to let it live a little longer.

hgy59 · 2024-03-09T11:11:04Z

since #6025 we don't have to use openssl 1.1.1 for ARMv5.

Regarding future versions of OpenSSL, there is no anouncement of the next LTS version yet.
The statement is, that every four years there will be an LTS version for at least 5 years (see OpenSSL Release Stategy)

In the past, there was an LTS version every about 3 years and my guess is, that there will be a new LTS version later this year:

LTS version	From	To	Comment
1.0.2	22.01.2015	20.12.2019	not an official LTS version
1.1.1	11.09.2018	11.09.2023
3.0	07.09.2021	07.09.2026
?	2024	2029	my guess

IMHO we don't need openssl 3.2 now, and should wait for the next LTS version (3.3, 4.0 ???).
Hopefully no package will depend on openssl 3.2 before the next LTS version is available.

And I think we should stay on OpenSSL 3.1 that is supported until 14.03.2025. I don't see any reason to go back to version 3.0. For the far future, we should provide only LTS versions of OpenSSL (it was my oversight when creating cross/openssl3 with v3.1).

BTW
I would avoid naming a dependency openssl-latest to allow a package by package migration when introducing a new latest version. (Since we had trouble updating python3 from 3.7 to 3.8, we solved this later by adding python310 and kept this scheme for python311).

publicarray · 2024-04-17T12:39:11Z

Using openssl 1.1 default with ⚠️ to community

I think adding a known vulnerable SSL lib is asking for trouble, people tend not to read warnings, If people want to compile it themselves that's fine.

I'm of the opinion that we should avoid knowingly publishing any package with CVEs (at the time of publishing). My 2c

https://www.openssl.org/news/vulnerabilities.html search for "premium support" e.g. Fixed in OpenSSL 1.1.1x (premium support)
Currently 3 vulnerabilities exists in 1.1.1

th0ma7 · 2024-04-17T20:09:10Z

Good point, and further as @hgy59 found a workaround for armv5. I'll mark this pr as closed.

th0ma7 requested a review from hgy59 January 28, 2024 15:20

th0ma7 self-assigned this Jan 28, 2024

th0ma7 added 3 commits January 28, 2024 15:25

openssl: Enable multi-openssl version (i.e. 1.1, 3.1 and latest)

45456b4

NOTE: Latest is not being used currently but builds ok.

openssl3: Convert to using default openssl

1649aa0

ARMv5: Re-enable support for armv5

a9f8eb0

th0ma7 force-pushed the multi-openssl branch from 3dde651 to a9f8eb0 Compare January 28, 2024 15:25

openssl-1.1: Re-introduce @hgy59 update to openssl-1.1.1w

5d80e29

openssl-3.x: Downgrade to 3.0.x to use LTS version

e9e584c

hgy59 removed their request for review April 17, 2024 22:58

publicarray closed this Apr 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support multiple openssl versions #5996

Support multiple openssl versions #5996

th0ma7 commented Jan 28, 2024 •

edited

Loading

th0ma7 commented Jan 28, 2024

th0ma7 commented Jan 28, 2024

hgy59 commented Jan 28, 2024 •

edited

Loading

th0ma7 commented Jan 28, 2024 •

edited

Loading

hgy59 commented Jan 28, 2024

hgy59 commented Jan 28, 2024

hgy59 commented Jan 28, 2024 •

edited

Loading

th0ma7 commented Jan 28, 2024 •

edited

Loading

hgy59 commented Jan 28, 2024

th0ma7 commented Jan 28, 2024

th0ma7 commented Jan 28, 2024

hgy59 commented Jan 28, 2024

th0ma7 commented Jan 28, 2024

hgy59 commented Feb 2, 2024

th0ma7 commented Feb 3, 2024

hgy59 commented Mar 9, 2024 •

edited

Loading

publicarray commented Apr 17, 2024 •

edited

Loading

th0ma7 commented Apr 17, 2024

Support multiple openssl versions #5996

Support multiple openssl versions #5996

Conversation

th0ma7 commented Jan 28, 2024 • edited Loading

Description

Checklist

Type of change

Related

How?

IMPORTANT

th0ma7 commented Jan 28, 2024

th0ma7 commented Jan 28, 2024

hgy59 commented Jan 28, 2024 • edited Loading

th0ma7 commented Jan 28, 2024 • edited Loading

hgy59 commented Jan 28, 2024

hgy59 commented Jan 28, 2024

hgy59 commented Jan 28, 2024 • edited Loading

th0ma7 commented Jan 28, 2024 • edited Loading

hgy59 commented Jan 28, 2024

th0ma7 commented Jan 28, 2024

th0ma7 commented Jan 28, 2024

hgy59 commented Jan 28, 2024

th0ma7 commented Jan 28, 2024

hgy59 commented Feb 2, 2024

th0ma7 commented Feb 3, 2024

hgy59 commented Mar 9, 2024 • edited Loading

publicarray commented Apr 17, 2024 • edited Loading

th0ma7 commented Apr 17, 2024

th0ma7 commented Jan 28, 2024 •

edited

Loading

hgy59 commented Jan 28, 2024 •

edited

Loading

th0ma7 commented Jan 28, 2024 •

edited

Loading

hgy59 commented Jan 28, 2024 •

edited

Loading

th0ma7 commented Jan 28, 2024 •

edited

Loading

hgy59 commented Mar 9, 2024 •

edited

Loading

publicarray commented Apr 17, 2024 •

edited

Loading