Added optional arg to -D/--podman/--docker base image to use #3869

judovana · 2024-06-28T16:40:55Z

it is also correctly passed to image generator, however taht is not tuned to work

#3855

judovana · 2024-06-28T16:42:03Z

I'm going to continue to work on this until build passes on fedora and rhel, but initial comments welcommed

docker/dockerfile-generator.sh

sbin/common/config_init.sh

sbin/common/common.sh

docker/dockerfile-generator.sh

judovana · 2024-07-01T13:40:08Z

Ok, it seems feature completed. Sorry for that reordering, but it speedup things really a lot.
It seems this is working on centos8/9 fedora39 (on f40 the gcc is to new) and default ubuntu. It seems ok on intel and aarch64!-)

There are still weird random failures solvable by --custom-cacerts . I hit it now on docker, with mkdir:

Generating cacerts from Mozilla's bundle
mkdir: cannot create directory 'certs': Permission denied

I could hjave casued that by reorder of steps. But I Did not moved the content behind USER build before it or before USER build behind it.

judovana · 2024-07-01T13:46:26Z

The failures seems to be related to eol of centos7

Actually, I had not tested this on centos7, will try

judovana · 2024-07-01T14:10:40Z

tested matrix (hw f40, amd64/aarch64, thusdefault is quay.io)

	podman jdk21u	docker jdk21u	podman jdk11u	docker jdk11u	podman jdk8u	docker jdk8u
aarch64 fedora:39	ok	n/a
x86_64 fedora:39	ok	ok
aarch64 centos:stream8	ok	n/a
x86_64 centos:stream8	ok	ok
aarch64 centos:stream9	ok	n/a
x86_64 centos:stream9	ok	ok
aarch64 ubuntu:18.0.4 (default)	ok	n/a
x86_64 ubuntu:18.0.4 (default)	ok	ok
aarch64 hw, x86_64 centos:7	failed[1]	n/a	failed late in build	n/a	failed[2]	n/a
x86_64 hw, x86_64 centos:7			failed late in build	faield late in build	ok
aarch64 hw, aarch64 docker.io/centos:7					ok	n/a


todo rhel7-9

[1] no boot jdk found, system fine
[2] system fine,  temurin jdk8 fails due to arch misaligned I guess

judovana · 2024-07-01T15:52:17Z

Ok, I consider testing table as finished and ok to go. Any prerequisites ion newer jdks on el7?

it is also correctly passed to image generator, however taht is not tuned to work

With this, the fedora containers finshed correctly with images

Co-authored-by: Martijn Verburg <[email protected]>

Generally sort them by disk footprint size. Motivation is, that allt he snapshjots were created after the install of all deps hapened. So instead of snapshoting 55mb base image with new symlink, it was snapshoting 1g or more image with all deps. Now all small-sized operations happens firsr then basic dependences happens, so most of the other operations can run last are huge ones: installing of all dependencies and cloning the jdk All is closed byu change of user to openjdk That reuired splitting of dn/apt to two parts, and thus extracting a bit of suddenly duplicated code to functions

docker/dockerfile-generator.sh

sbin/common/config_init.sh

Co-authored-by: Martijn Verburg <[email protected]>

judovana · 2024-07-02T06:55:11Z

I had dropped that comment. It is needed. I will consult with more rhel jdk developers thogh (but it may take quite a time)

sxa · 2024-07-02T09:31:45Z

docker/dockerfile-generator.sh

+  if echo "${IMAGE}" | grep -e "stream8" -e "centos:7" ; then
+    echo " 
+RUN cd /etc/yum.repos.d/ ; sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-*
+RUN cd /etc/yum.repos.d/ ; sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-* " >> "$DOCKERFILE_PATH"


This is going to need a follow-on change as the non-x64 ones (aarch64 and ppc64le) are under vault.centos.org/altarch (This wasn't a problem for the CentOS6 ones we translated in the past as we only used that one on x64)

@steelhead31 is working on a fix elsewhere as he hit the same problem after merging adoptium/infrastructure#3643 so once he's completed that we can replicate the same operation in here.

That sould be simple check, but I was trying both centos:stream{8,9} on aarch64. And centos8 seems to be already facing the "need of sed" issue. I will double check. I could have had swapped VM.

If it indeed will not run, I will adjust, shoudl be simple. TYVM!

Really, it works for8:

tester@fedora:~/temurin-build$ sh makejdk-any-platform.sh -D centos:stream8 --custom-cacerts jdk21u Starting makejdk-any-platform.sh to configure, build (Adoptium)OpenJDK binary Parsing opt: -D Possible opt arg: centos:stream8 base image will be set to: centos:stream8 Parsing opt: --custom-cacerts Possible opt arg: jdk21u Setting version based on forest_name=jdk21u Default branch set to BUILD_CONFIG[BRANCH]=master Using source repository https://github.com/openjdk/jdk21u Working dir is ./build/ [debug] COPY_MACOSX_FREE_FONT_LIB_FOR_JDK_FLAG=false [debug] COPY_MACOSX_FREE_FONT_LIB_FOR_JRE_FLAG=false JDK Image folder name: jdk JRE Image folder name: jre Removing Dockerfile* from docker/jdk21/aarch64/centos centos:stream8 centos:stream8 centos:stream8 Dockerfile created at docker/jdk21/aarch64/centos/Dockerfile ... --> c9d0997cfc9c STEP 18/42: RUN chmod 755 /home/tester/temurin-build/workspace/config --> 0034495152dd STEP 19/42: RUN chown -R build /home/tester/temurin-build/workspace/config --> 2754a7505488 STEP 20/42: RUN cd /etc/yum.repos.d/ ; sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-* --> ce5684c8a2eb STEP 21/42: RUN cd /etc/yum.repos.d/ ; sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-* --> a6ed67c4dbcc STEP 22/42: RUN dnf -y update --allowerasing CentOS Stream 8 - AppStream 22 MB/s | 25 MB 00:01 CentOS Stream 8 - BaseOS 11 MB/s | 7.0 MB 00:00 CentOS Stream 8 - Extras 39 kB/s | 17 kB 00:00 CentOS Stream 8 - Extras common packages 18 kB/s | 8.0 kB 00:00 Dependencies resolved. ================================================================================ Package Architecture Version Repository Size ================================================================================ Upgrading: hostname aarch64 3.20-7.el8.0.1 baseos 32 k Transaction Summary ================================================================================ Upgrade 1 Package Total download size: 32 k Downloading Packages: hostname-3.20-7.el8.0.1.aarch64.rpm 45 kB/s | 32 kB 00:00 -------------------------------------------------------------------------------- Total 45 kB/s | 32 kB 00:00 CentOS Stream 8 - BaseOS 1.6 MB/s | 1.6 kB 00:00 Importing GPG key 0x8483C65D: Userid : "CentOS (CentOS Official Signing Key) <[email protected]>" ... Verifying : hostname-3.20-6.el8.aarch64 2/2 Upgraded: hostname-3.20-7.el8.0.1.aarch64 Complete! --> f57ef513cfff STEP 23/42: RUN dnf -y install --allowerasing bzip2-libs bzip2 curl git unzip wget zip Last metadata expiration check: 0:00:09 ago on Tue Jul 2 10:00:52 2024. Package bzip2-libs-1.0.6-26.el8.aarch64 is already installed. Package curl-7.61.1-34.el8.aarch64 is already installed. Dependencies resolved. ======================================================================================= Package Arch Version Repo Size ======================================================================================= Installing: bzip2 aarch64 1.0.6-26.el8 baseos 60 k git aarch64 2.43.0-1.el8 appstream 92 k unzip aarch64 6.0-46.el8 baseos 189 k wget aarch64 1.19.5-11.el8 appstream 717 k zip aarch64 3.0-23.el8 baseos 265 k Installing dependencies: ... our archive was created as /home/tester/temurin-build/workspace/build/src/build/linux-aarch64-server-release/images/OpenJDK.tar.gz Moving the artifact to location /home/tester/temurin-build/workspace/target//OpenJDK-static-libs-glibc.tar.gz archive done. Archiving and compressing with gzip gzip: warning: GZIP environment variable is deprecated; use an alias or script Your archive was created as /home/tester/temurin-build/workspace/build/src/build/linux-aarch64-server-release/images/OpenJDK.tar.gz Moving the artifact to location /home/tester/temurin-build/workspace/target//OpenJDK-jdk.tar.gz real 0m49.688s user 0m48.709s sys 0m1.121s archive done. Skip generating SBOM build.sh : 10:19:01 : All done! The finished image can be found in /home/tester/temurin-build/workspace/target on the host system Removing container jdk21-hotspot 3d0f00425bbf

The alt-arch was only cento7 speciality. So I guess you were referring to it. Sorry, I misread a bit. I will try alsocentos7 on aarch64, and I guess thatr will need the alt for sure.

Again, my solution is pretty similar.. wrapped in an if statement..

sed -i 's|#baseurl=http://mirror.centos.org/altarch/\$releasever/|baseurl=http://vault.centos.org/altarch/7.9.2009/|' /etc/yum.repos.d/CentOS-Base.repo; \

RUN sed -i -e 's!mirrorlist!#mirrorlist!g' /etc/yum.repos.d/CentOS-Base.repo RUN if \[ "$(uname -m)" = "aarch64" \]; then \\ echo "Running on arm64 architecture"; \\ sed -i 's|#baseurl=http://mirror.centos.org/altarch/\$releasever/|baseurl=http://vault.centos.org/altarch/7.9.2009/|' /etc/yum.repos.d/CentOS-Base.repo; \\ elif \[ "$(uname -m)" = "ppc64le" \]; then \\ echo "Running on ppc64le architecture"; \\ sed -i 's|#baseurl=http://mirror.centos.org/altarch/\$releasever/|baseurl=http://vault.centos.org/altarch/7.9.2009/|' /etc/yum.repos.d/CentOS-Base.repo; \\ else \\ # Default To x64; \\ echo "Running on another architecture"; \\ sed -i -e 's!#baseurl=http://mirror.centos.org/centos/\$releasever!baseurl=https://vault.centos.org/7.9.2009/!g' /etc/yum.repos.d/CentOS-Base.repo; \\ fi

Again, my solution is pretty similar.. wrapped in an if statement..

sed -i 's|#baseurl=http://mirror.centos.org/altarch/\$releasever/|baseurl=http://vault.centos.org/altarch/7.9.2009/|' /etc/yum.repos.d/CentOS-Base.repo; \

RUN sed -i -e 's!mirrorlist!#mirrorlist!g' /etc/yum.repos.d/CentOS-Base.repo RUN if \[ "$(uname -m)" = "aarch64" \]; then \\ echo "Running on arm64 architecture"; \\ sed -i 's|#baseurl=http://mirror.centos.org/altarch/\$releasever/|baseurl=http://vault.centos.org/altarch/7.9.2009/|' /etc/yum.repos.d/CentOS-Base.repo; \\ elif \[ "$(uname -m)" = "ppc64le" \]; then \\ echo "Running on ppc64le architecture"; \\ sed -i 's|#baseurl=http://mirror.centos.org/altarch/\$releasever/|baseurl=http://vault.centos.org/altarch/7.9.2009/|' /etc/yum.repos.d/CentOS-Base.repo; \\ else \\ # Default To x64; \\ echo "Running on another architecture"; \\ sed -i -e 's!#baseurl=http://mirror.centos.org/centos/\$releasever!baseurl=https://vault.centos.org/7.9.2009/!g' /etc/yum.repos.d/CentOS-Base.repo; \\ fi

Thanx!! I will reuse it immediately if it bites back or if @sxa insits.

judovana · 2024-07-02T10:33:09Z

@sxa mm... it works also with centos:7!

  makejdk-any-platform.sh   -D centos:7   --custom-cacerts  jdk21u
Starting makejdk-any-platform.sh to configure, build (Adoptium)OpenJDK binary
Parsing opt: -D
Possible opt arg: centos:7
base image will be set to: centos:7
Parsing opt: --custom-cacerts
Possible opt arg: jdk21u
Setting version based on forest_name=jdk21u
Default branch set to BUILD_CONFIG[BRANCH]=master
Using source repository https://github.com/openjdk/jdk21u
Working dir is ./build/
[debug] COPY_MACOSX_FREE_FONT_LIB_FOR_JDK_FLAG=false
[debug] COPY_MACOSX_FREE_FONT_LIB_FOR_JRE_FLAG=false
JDK Image folder name: jdk
JRE Image folder name: jre
Removing Dockerfile* from docker/jdk21/aarch64/centos
centos:7
centos:7
centos:7
centos:7
centos:7
Dockerfile created at docker/jdk21/aarch64/centos/Dockerfile
Using Docker to build the JDK
...
STEP 19/42: RUN chown -R build /home/tester/temurin-build/workspace/config
--> Using cache d8ef18f000c0177244eab45c27a646957b56d0175b7c051120f4c76ec03f3643
--> d8ef18f000c0
STEP 20/42: RUN cd /etc/yum.repos.d/ ; sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-*
--> 9a4fb9eb3a8e
STEP 21/42: RUN cd /etc/yum.repos.d/ ; sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-* 
--> 98fc1c34adfc
STEP 22/42: RUN yum  -y update 
Loaded plugins: fastestmirror, ovl
Determining fastest mirrors
Resolving Dependencies
--> Running transaction check
---> Package bash.x86_64 0:4.2.46-34.el7 will be updated
---> Package bash.x86_64 0:4.2.46-35.el7_9 will be an update
...
---> Package xz-libs.x86_64 0:5.2.2-2.el7_9 will be an update
---> Package zlib.x86_64 0:1.2.7-18.el7 will be updated
---> Package zlib.x86_64 0:1.2.7-21.el7_9 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package              Arch     Version                          Repository
                                                                           Size
================================================================================
Updating:
 bash                 x86_64   4.2.46-35.el7_9                  updates   1.0 M
 bind-license         noarch   32:9.11.4-26.P2.el7_9.16         updates    92 k
 binutils             x86_64   2.27-44.base.el7_9.1             updates   5.9 M
...
 xz                   x86_64   5.2.2-2.el7_9                    updates   229 k
 xz-libs              x86_64   5.2.2-2.el7_9                    updates   103 k
 zlib                 x86_64   1.2.7-21.el7_9                   updates    90 k

Transaction Summary
================================================================================
Upgrade  51 Packages

Total download size: 53 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
warning: /var/cache/yum/x86_64/7/updates/packages/bind-license-9.11.4-26.P2.el7_9.16.noarch.rpm: Header V4 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Public key for bind-license-9.11.4-26.P2.el7_9.16.noarch.rpm is not installed
--------------------------------------------------------------------------------
Total                                              4.4 MB/s |  53 MB  00:11     
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Importing GPG key 0xF4A80EB5:
 Userid     : "CentOS-7 Key (CentOS 7 Official Signing Key) <[email protected]>"
 Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
 Package    : centos-release-7-9.2009.0.el7.centos.x86_64 (@CentOS)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : centos-release-7-9.2009.2.el7.centos.x86_64                1/102git config pull.rebase true   
  Updating   : tzdata-2024a-1.el7.noarch                                  2/102 
  Updating   : bash-4.2.46-35.el7_9.x86_64                                3/102 
  Updating   : nss-softokn-freebl-3.90.0-6.el7_9.x86_64                   4/102

judovana · 2024-07-02T11:00:40Z

@sxa see #3869 (comment)

it seems centos7 is ok. Is there anything missing to yuou in that table?

judovana · 2024-07-02T11:20:03Z

@sxa see #3869 (comment)

it seems centos7 is ok. Is there anything missing to yuou in that table?

Gosh. that centos7 thing is pulling x86_64 packages.... Sorry,. overlooked that

judovana · 2024-07-02T14:17:24Z

Because it is intel image! WARNING: image platform (linux/amd64) does not match the expected platform (linux/arm64) my vm went wild.

judovana · 2024-07-02T15:21:17Z

@sxa ...so after double-checked the architectures (what a burned 3hours!.. but I got my lesson) It reallyh seems I do not need to sed the repos . All deps were pulled and jdk 8 built fine. I removed all images and trying again.

sxa

On the basis that the testing looks good, and this will not have an impact on the Production builds I'm happy to approve. I will note that we should look at updating from the default of an out-of-support Ubuntu 18.04 at some point for the container.

judovana · 2024-07-03T14:31:02Z

@sxa I think the Ubuntu 18,0,4 is still supported...

edit: I misunderstood the page, ubuntu 18.0.4 isd no longer freely supported. It i only behind payed fairewall. (@karianna so I misinformed also you, sorry, it was not intentional, I got that support model wrong)

github-actions bot added the docker Issues related to our docker files and docker scripts label Jun 28, 2024

judovana force-pushed the rpmContainer branch 3 times, most recently from 5058bd8 to a7c0a75 Compare June 30, 2024 19:07

karianna requested changes Jul 1, 2024

View reviewed changes

judovana force-pushed the rpmContainer branch 2 times, most recently from 1e39878 to 84ab0b2 Compare July 1, 2024 06:35

judovana and others added 15 commits July 2, 2024 17:09

Added optional arg to -D/--podman/--docker base image to use

0f2fbe8

it is also correctly passed to image generator, however taht is not tuned to work

seaparate local declaration and no backticks and no sed

0a158ed

Using current arch as arch instead of ahrdcoded

205b604

Added fedora depndencies

0ddc037

With this, the fedora containers finshed correctly with images

Now working also on centos 9

27d92c8

Now centos:stream8

67936fb

Fixed indentation

52213c3

Update sbin/common/common.sh

5826bcd

Co-authored-by: Martijn Verburg <[email protected]>

missing article

bb9b983

Co-authored-by: Martijn Verburg <[email protected]>

Added more coments to optional image argumetn check

482e4df

Removed hardcoded x64 arch

e2c3935

Added few more commens if comments are on

b1a436b

$() and " for bash lint

5670083

Added support for centos:7

03a514f

karianna force-pushed the rpmContainer branch from 545006e to 03a514f Compare July 2, 2024 05:09

karianna requested changes Jul 2, 2024

View reviewed changes

docker/dockerfile-generator.sh Outdated Show resolved Hide resolved

sbin/common/config_init.sh Outdated Show resolved Hide resolved

sbin/common/config_init.sh Outdated Show resolved Hide resolved

sbin/common/config_init.sh Outdated Show resolved Hide resolved

Update sbin/common/config_init.sh

10a27ff

Co-authored-by: Martijn Verburg <[email protected]>

judovana and others added 3 commits July 2, 2024 08:19

Update sbin/common/config_init.sh

990b358

Co-authored-by: Martijn Verburg <[email protected]>

Update sbin/common/config_init.sh

3320ef9

Co-authored-by: Martijn Verburg <[email protected]>

removed missleading comment

746bdc4

judovana requested a review from karianna July 2, 2024 06:55

sxa reviewed Jul 2, 2024

View reviewed changes

sxa approved these changes Jul 3, 2024

View reviewed changes

sxa requested a review from andrew-m-leonard July 3, 2024 14:18

judovana requested a review from steelhead31 July 3, 2024 17:15

karianna approved these changes Jul 4, 2024

View reviewed changes

karianna merged commit 2c76461 into adoptium:master Jul 4, 2024
20 of 27 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added optional arg to -D/--podman/--docker base image to use #3869

Added optional arg to -D/--podman/--docker base image to use #3869

judovana commented Jun 28, 2024 •

edited

Loading

judovana commented Jun 28, 2024

judovana commented Jul 1, 2024

judovana commented Jul 1, 2024

judovana commented Jul 1, 2024 •

edited

Loading

judovana commented Jul 1, 2024

judovana commented Jul 2, 2024

sxa Jul 2, 2024 •

edited

Loading

judovana Jul 2, 2024

judovana Jul 2, 2024 •

edited

Loading

judovana Jul 2, 2024

steelhead31 Jul 2, 2024 •

edited

Loading

judovana Jul 2, 2024

judovana commented Jul 2, 2024

judovana commented Jul 2, 2024

judovana commented Jul 2, 2024

judovana commented Jul 2, 2024

judovana commented Jul 2, 2024

sxa left a comment

judovana commented Jul 3, 2024 •

edited

Loading

Added optional arg to -D/--podman/--docker base image to use #3869

Added optional arg to -D/--podman/--docker base image to use #3869

Conversation

judovana commented Jun 28, 2024 • edited Loading

judovana commented Jun 28, 2024

judovana commented Jul 1, 2024

judovana commented Jul 1, 2024

judovana commented Jul 1, 2024 • edited Loading

judovana commented Jul 1, 2024

judovana commented Jul 2, 2024

sxa Jul 2, 2024 • edited Loading

Choose a reason for hiding this comment

judovana Jul 2, 2024

Choose a reason for hiding this comment

judovana Jul 2, 2024 • edited Loading

Choose a reason for hiding this comment

judovana Jul 2, 2024

Choose a reason for hiding this comment

steelhead31 Jul 2, 2024 • edited Loading

Choose a reason for hiding this comment

judovana Jul 2, 2024

Choose a reason for hiding this comment

judovana commented Jul 2, 2024

judovana commented Jul 2, 2024

judovana commented Jul 2, 2024

judovana commented Jul 2, 2024

judovana commented Jul 2, 2024

sxa left a comment

Choose a reason for hiding this comment

judovana commented Jul 3, 2024 • edited Loading

judovana commented Jun 28, 2024 •

edited

Loading

judovana commented Jul 1, 2024 •

edited

Loading

sxa Jul 2, 2024 •

edited

Loading

judovana Jul 2, 2024 •

edited

Loading

steelhead31 Jul 2, 2024 •

edited

Loading

judovana commented Jul 3, 2024 •

edited

Loading