Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Try to get Gentoo to provide needed files on the distmirrors #2

Open
NiKiZe opened this issue Jul 18, 2021 · 9 comments
Open

Try to get Gentoo to provide needed files on the distmirrors #2

NiKiZe opened this issue Jul 18, 2021 · 9 comments
Labels
help wanted Extra attention is needed

Comments

@NiKiZe
Copy link
Owner

NiKiZe commented Jul 18, 2021

This is

  • gentoo kernel
  • gentoo.igz original initrd
  • image.squashfs
  • boot.ipxe the iPXE script - once it is well tested for all platforms
@NiKiZe
Copy link
Owner Author

NiKiZe commented Jul 18, 2021

This is a discussion with an already existing Bug, probably not to hard to get fix once we have good documentation here on how it could be used. https://bugs.gentoo.org/494300

@NiKiZe
Copy link
Owner Author

NiKiZe commented Jul 21, 2021

From chat using https://web.libera.chat/#gentoo-releng

jpds
NiKiZe: [as a user]: I'd like that, debian has the stuff under http://ftp.uk.debian.org/debian/dists/stable/main/installer-amd64/current/images/netboot/debian-installer/amd64/

NiKiZe
jpds yes, the current iso works as it is, so I have not found any reason to make it netboot specific? But I guess this somewhat boils down to maintenance and automation. If anyone could point me in the right direction for where minimal iso is created and copied, then maybe I could provide a patch to move forward?

iamben
look in releng.git repo, tools/catalyst-auto calls catalyst, uploading happens via the arch-specific catalyst-auto-*.conf files
im not sure if it would be best implemented in catalyst-auto, or as a separate build target in catalyst itself

NiKiZe
Thanks! Since the artifacts should already exist from building the iso I'm assuming it's best to reuse that if possible? Could you explain just a little more where I should look for "separate build target in catalyst"?

iamben
https://github.com/gentoo/catalyst/tree/master/targets

gentoo/releng
https://github.com/gentoo/releng/blob/master/tools/catalyst-auto-amd64-demeter.conf
https://wiki.gentoo.org/wiki/Catalyst

@NiKiZe NiKiZe pinned this issue Jul 21, 2021
NiKiZe referenced this issue in NiKiZe/systemrescuecd Jul 22, 2021
@NiKiZe
Copy link
Owner Author

NiKiZe commented Jul 22, 2021

Notes for trying to use catalyst

mkdir -p /etc/portage/package.accept_keywords
echo dev-util/catalyst > /etc/portage/package.accept_keywords/catalyst
echo 'GRUB_PLATFORMS="${GRUB_PLATFORMS} emu efi-32 efi-64 pc"' >> /etc/portage/make.conf
#-boot to avoid boot partition issue with memtest
#system-bootloader needed for livecd creation
USE="-boot system-bootloader" emerge -uvN pixz catalyst genlop reptyr -j16
# python mount needed (maybe)
USE=python emerge -uvN util-linux

# grep -v ^# /etc/catalyst/catalyst.conf
# remove autoresume
# add preserve_libs
# add keepwork
# remove snapcache - first build fails with this enabled ?
options="bindist kerncache pkgcache preserve_libs keepwork seedcache"
snapshot_cache="/release/tmp/snapshots"
storedir="/release/buildroot/amd64-dev"
source_matching="loose"
# add log
port_logdir="/var/tmp/catalyst/tmp"
var_tmpfs_portage=16

# cat /etc/catalyst/catalystrc
export MAKEOPTS="-j8"
export EMERGE_DEFAULT_OPTS="--jobs 20 --load-average 9"


cd
git clone https://github.com/gentoo/releng.git
cd releng

# TODO parse /etc/catalyst/catalyst.conf to get below paths

mkdir -p /release/buildroot/amd64-dev/builds/default/
mkdir -p /release/buildroot/amd64-dev/builds/hardened/
mkdir -p /release/buildroot/amd64-dev/builds/musl
mkdir -p /release/buildroot/amd64-dev/builds/musl-hardened
mkdir -p /release/tmp/snapshots/
pushd /release/buildroot/amd64-dev/builds/default/
FILE=$(wget -q http://distfiles.gentoo.org/releases/amd64/autobuilds/current-install-amd64-minimal/ -O - | grep -o -E 'stage3-amd64-openrc-20\w*\.tar\.(bz2|xz)' | uniq)
wget http://distfiles.gentoo.org/releases/amd64/autobuilds/current-install-amd64-minimal/$FILE
# ln -s $FILE stage3-amd64-openrc-latest.tar.xz
popd

# check expected times:
genlop -t gcc

mkdir -p /release/buildroot/amd64-dev/tmp/; \
umount /release/buildroot/amd64-dev/tmp/; \
rm -rf /release/buildroot/amd64-dev/tmp/*; mount none -t tmpfs /release/buildroot/amd64-dev/tmp/ -o size=16G,nr_inodes=1M; \
sh tools/catalyst-auto -v -v -X -l /var/lock/catalyst.lock -c tools/catalyst-auto-amd64-demeter.conf; \
umount /release/buildroot/amd64-dev/tmp/

@NiKiZe
Copy link
Owner Author

NiKiZe commented Jul 23, 2021

Building with snapcache enabled gives this:

*** Running command: catalyst -a -c /etc/catalyst/catalyst.conf -f /tmp/catalyst-auto.20210723T062037Z.zsSPDX/specs/stage1-openrc.spec
23 Jul 2021 08:20:44 CEST: NOTICE  : Loading configuration file: /etc/catalyst/catalyst.conf
23 Jul 2021 08:20:44 CEST: NOTICE  : conf_values[options] = {'preserve_libs', 'pkgcache', 'clear-autoresume', 'keepwork', 'seedcache', 'snapcache', 'bindist', 'kerncache'}
23 Jul 2021 08:20:44 CEST: NOTICE  : Processing spec file: /tmp/catalyst-auto.20210723T062037Z.zsSPDX/specs/stage1-openrc.spec
23 Jul 2021 08:20:44 CEST: NOTICE  : Using target: stage1
23 Jul 2021 08:20:44 CEST: NOTICE  : Source file specification matching setting is: loose
23 Jul 2021 08:20:44 CEST: NOTICE  : Accepted source file extensions search order: ['tar.xz', 'tpxz', 'xz', 'tar.bz2', 'bz2', 'tbz2', 'squashfs', 'sfs', 'tar.gz', 'gz', 'tar']
23 Jul 2021 08:20:44 CEST: NOTICE  : Source path set to /release/buildroot/amd64-dev/builds/default/stage3-amd64-openrc-latest.tar.xz
23 Jul 2021 08:20:44 CEST: NOTICE  : stage1 stage path is /release/buildroot/amd64-dev/tmp/default/stage1-amd64-openrc-20210723T062037Z/tmp/stage1root
23 Jul 2021 08:20:44 CEST: NOTICE  : --- Running action sequence: unpack
23 Jul 2021 08:20:44 CEST: NOTICE  : Referenced SEEDCACHE does not appear to be a directory, trying to untar...
23 Jul 2021 08:20:44 CEST: NOTICE  : Clearing the chroot path ...
23 Jul 2021 08:20:44 CEST: NOTICE  : Emptying directory: /release/buildroot/amd64-dev/tmp/default/stage1-amd64-openrc-20210723T062037Z
23 Jul 2021 08:20:44 CEST: NOTICE  : Starting auto from /release/buildroot/amd64-dev/builds/default/stage3-amd64-openrc-latest.tar.xz
23 Jul 2021 08:20:44 CEST: NOTICE  : to /release/buildroot/amd64-dev/tmp/default/stage1-amd64-openrc-20210723T062037Z (this may take some     time) ..
23 Jul 2021 08:20:47 CEST: NOTICE  : --- Running action sequence: unpack_snapshot
23 Jul 2021 08:20:47 CEST: NOTICE  : Unpacking portage tree (this can take a long time) ...
tar: /release/tmp/snapshots/20210723T062037Z: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
23 Jul 2021 08:20:47 CEST: ERROR   : Error unpacking snapshot using mode lbzip2
23 Jul 2021 08:20:47 CEST: ERROR   : Exception running action sequence unpack_snapshot
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/catalyst/base/stagebase.py", line 1499, in _run
    getattr(self, x)()
  File "/usr/lib/python3.9/site-packages/catalyst/base/stagebase.py", line 891, in unpack_snapshot
    with open(snapshot_cache_hash_path, 'w') as myf:
FileNotFoundError: [Errno 2] No such file or directory: '/release/tmp/snapshots/20210723T062037Z/catalyst-hash'
23 Jul 2021 08:20:47 CEST: NOTICE  : Cleaning up... Running unbind()

@NiKiZe
Copy link
Owner Author

NiKiZe commented Jul 24, 2021

bootstrap stage2 fails, it was ok first time, but even after clearing all binpkgs still getting this.
disabling binpkgs might be needed if there is any pkgs left around at all

Copying stage2-chroot.sh to /tmp
copying stage2-chroot.sh to /release/buildroot/amd64-dev/tmp/default/stage2-amd64-openrc-20210724T051742Z/tmp
copying chroot-functions.sh to /release/buildroot/amd64-dev/tmp/default/stage2-amd64-openrc-20210724T051742Z/tmp
Ensure the file has the executable bit set
Running stage2-chroot.sh in chroot:
    chroot /release/buildroot/amd64-dev/tmp/default/stage2-amd64-openrc-20210724T051742Z /tmp/stage2-chroot.sh
>>> Regenerating /etc/ld.so.cache...

Gentoo Linux; http://www.gentoo.org/
Copyright 1999-2020 Gentoo Authors; Distributed under the GPLv2
-------------------------------------------------------------------------------
  [[ (0/3) Locating packages ]]
 * Using baselayout : >=sys-apps/baselayout-2
 * Using portage    : sys-apps/portage
 * Using os-headers : >=sys-kernel/linux-headers-5.10
 * Using binutils   : sys-devel/binutils
 * Using gcc        : sys-devel/gcc
 * Using gettext    : sys-devel/gettext
 * Using libc       : sys-libs/glibc:2.2
 * Using texinfo    : sys-apps/texinfo
 * Using zlib       : sys-libs/zlib
 * Using ncurses    : sys-libs/ncurses
-------------------------------------------------------------------------------
  [[ (1/3) Configuring environment ]]
-------------------------------------------------------------------------------
  [[ (2/3) Updating portage ]]
>>> Running pre-merge checks for sys-apps/portage-3.0.20-r6
 * Checking for suitable kernel configuration options...
 [ ok ]
>>> Emerging binary (1 of 1) sys-apps/portage-3.0.20-r6::gentoo
>>> Installing (1 of 1) sys-apps/portage-3.0.20-r6::gentoo
-------------------------------------------------------------------------------
  [[ (3/3) Emerging packages ]]
[ebuild   R   ] sys-apps/baselayout-2.7
[ebuild   R   ] sys-libs/zlib-1.2.11-r4
[ebuild   R   ] sys-kernel/linux-headers-5.10
[ebuild   R   ] sys-devel/gettext-0.21-r1
[ebuild   R   ] sys-devel/binutils-2.35.2
[ebuild   R   ] sys-devel/gcc-10.3.0-r2
[ebuild   R   ] sys-libs/glibc-2.33-r1
[ebuild  N    ] virtual/perl-IO-1.460.0
[ebuild  N    ] dev-perl/Unicode-EastAsianWidth-1.330.0-r1
[ebuild  N    ] dev-perl/Text-Unidecode-1.300.0
[ebuild  N    ] dev-perl/Params-Util-1.70.0
[ebuild  N    ] dev-perl/Class-Inspector-1.360.0
[ebuild  N    ] dev-perl/File-ShareDir-Install-0.130.0
[ebuild  N    ] dev-perl/Exporter-Tiny-1.2.2
[ebuild  N    ] dev-perl/List-MoreUtils-0.428.0
[ebuild  N    ] dev-perl/File-ShareDir-1.116.0
[ebuild  N    ] dev-perl/libintl-perl-1.310.0
[ebuild   R   ] sys-apps/texinfo-6.7

!!! Multiple package instances within a single package slot have been pulled
!!! into the dependency graph, resulting in a slot conflict:

sys-devel/gcc:10

  (sys-devel/gcc-10.3.0-r2:10/10::gentoo, ebuild scheduled for merge) USE="(cxx) (multilib) nls nptl (pie) (-ada) -d -debug -doc (-fixed-point) -fortran -go -graphite (-hardened) -jit (-libssp) -lto -objc -objc++ -objc-gc -openmp -pch -pgo -sanitize -ssp -systemtap -test -vanilla -vtv -zstd" ABI_X86="(64)" pulled in by
    sys-devel/gcc (Argument)

  (sys-devel/gcc-10.3.0-r2-1:10/10::gentoo, installed) USE="(cxx) fortran (multilib) nls nptl openmp pch (pie) sanitize ssp (-ada) -d -debug -doc (-fixed-point) -go -graphite (-hardened) -jit (-libssp) -lto -objc -objc++ -objc-gc -pgo -systemtap -test -vanilla -vtv -zstd" ABI_X86="(64)" pulled in by
    >=sys-devel/gcc-4.2:*[openmp] required by (app-crypt/libb2-0.98.1-r2-1:0/0::gentoo, installed) USE="openmp -native-cflags -static-libs" ABI_X86="(64) -32 (-x32)"
                          ^^^^^^                                                                                                                 
    >=sys-devel/gcc-4.2:*[openmp] required by (app-portage/portage-utils-0.90-1:0/0::gentoo, installed) USE="nls openmp qmanifest qtegrity -static" ABI_X86="(64)"
                          ^^^^^^                                                                                                                 


It might be possible to solve this slot collision
by applying all of the following changes:
   - sys-devel/gcc-10.3.0-r2 (Change USE: +openmp)

24 Jul 2021 07:22:22 CEST: ERROR   : CatalystError: cmd(['/usr/share/catalyst/targets/stage2/stage2-controller.sh', 'run']) exited 1
ERROR:catalyst:CatalystError: cmd(['/usr/share/catalyst/targets/stage2/stage2-controller.sh', 'run']) exited 1
24 Jul 2021 07:22:22 CEST: ERROR   : CatalystError: Stage build aborting due to error.
ERROR:catalyst:CatalystError: Stage build aborting due to error.
24 Jul 2021 07:22:22 CEST: ERROR   : Exception running action sequence run_local

@NiKiZe
Copy link
Owner Author

NiKiZe commented Jul 24, 2021

Testing if using this stage3 works around the openmp issue ...
Not working, but first test did upgrade of stage3 before using it as base + using binpkgs

FILE=$(wget -q http://distfiles.gentoo.org/releases/amd64/autobuilds/current-stage3-amd64/ -O - | grep -o -E 'stage3-amd64-20\w*\.tar\.(bz2|xz)' | uniq)
wget http://distfiles.gentoo.org/releases/amd64/autobuilds/current-stage3-amd64/$FILE
ln -s $FILE stage3-amd64-openrc-latest.tar.xz

@NiKiZe
Copy link
Owner Author

NiKiZe commented Jul 24, 2021

netboot targets exists that generates multiple files.
installcd-stage2-minimal.spec could probably be made to copy out needed files as well.

The cleanest way would be to split installcd-stage2-minimal.spec up into 2 files to get a stage2 and 3

  • installcd-stage2.spec would build squashfs, kernel, initrd.
  • installcd-stage3-minimal.spec would need to add bootloaders (current grub, syslinux), and generate iso.
  • installpxe-stage3-minimal.spec Would just repackage initrd+squashfs and possibly generate config files/scripts, but this still needs to generate multiple outputs?

However this probably makes to much of a disconnect between settings in the stage2 spec files and what would be needed for stage3. - not to mention all the places and documentation that would need updates in regards to livcd stage2/3.

in catalyst/targets/livecd_stage2.py function create_iso is added, which is defined in catalyst/base/stagebase.py and calls the targets/livecd-stage2/livecd-stage2-controller.sh with iso argument.
Maybe the needed copy out is best placed in targets/targets/support/create-iso.sh?
Or in the iso step of targets/livecd-stage2/livecd-stage2-controller.sh
(consider that we might want to create "gentoo.ipxe", and copy out and maybe modify both "grub.cfg" and "isolinux.cfg" (as pxelinux.cfg/default)
All this requires modification and release of catalyst.

In the end maybe the best, or at least the first solution, would be to simply extract the iso in the upload step instead?

@NiKiZe NiKiZe added the help wanted Extra attention is needed label Jul 24, 2021
NiKiZe added a commit to NiKiZe/releng that referenced this issue Jul 28, 2021
Extracts and generates needed files to perform iPXE boot from gentoo mirrors
Provided as PoC of what is needed for this to be vaild
If merged this would resolve NiKiZe/Gentoo-iPXE#2
Will be providing PoC for doing most of this in catalyst instead as well
Really want to have this in the same target as minimal livecd

To test result with qemu localy after build (replace your variables):
qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -nic user,model=virtio,\
tftp="${BUILD_SRCDIR_BASE}"/builds/default,\
bootfile=install-amd64-minimal-${TIMESTAMP}.ipxe \
-watchdog i6300esb -watchdog-action reset -boot menu=on -usb

Signed-off-by: Christian Nilsson <[email protected]>
NiKiZe added a commit to NiKiZe/releng that referenced this issue Jul 28, 2021
Extracts and generates needed files to perform iPXE boot from gentoo mirrors
Provided as PoC of what is needed for this to be vaild
If merged this would resolve NiKiZe/Gentoo-iPXE#2
Will be providing PoC for doing most of this in catalyst instead as well
Really want to have this in the same target as minimal livecd

To test result with qemu localy after build (replace your variables):
qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -nic user,model=virtio,\
tftp="${BUILD_SRCDIR_BASE}"/builds/default,\
bootfile=install-amd64-minimal-${TIMESTAMP}.ipxe \
-watchdog i6300esb -watchdog-action reset -boot menu=on -usb

Signed-off-by: Christian Nilsson <[email protected]>
NiKiZe added a commit to NiKiZe/catalyst that referenced this issue Jul 28, 2021
Copy out needed files from iso to perform iPXE boot from gentoo mirrors
Provided as PoC of what is needed for this to be vaild
If merged this would resolve NiKiZe/Gentoo-iPXE#2
There is an alternative aproch to extract and generate these files during upload instead
Really want to have this in the same target as minimal livecd

To test result with qemu localy after build (replace your variables):
qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -nic user,model=virtio,\
tftp="${BUILD_SRCDIR_BASE}"/builds/default,\
bootfile=install-amd64-minimal-${TIMESTAMP}.ipxe \
-watchdog i6300esb -watchdog-action reset -boot menu=on -usb

Signed-off-by: Christian Nilsson <[email protected]>
NiKiZe added a commit to NiKiZe/catalyst that referenced this issue Jul 29, 2021
Copy out needed files from iso to perform iPXE boot from gentoo mirrors
Provided as PoC of what is needed for this to be vaild
If merged this would resolve NiKiZe/Gentoo-iPXE#2
There is an alternative aproch to extract and generate these files during upload instead
Really want to have this in the same target as minimal livecd

To test result with qemu localy after build (replace your variables):
qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -nic user,model=virtio,\
tftp="${BUILD_SRCDIR_BASE}"/builds/default,\
bootfile=install-amd64-minimal-${TIMESTAMP}.ipxe \
-watchdog i6300esb -watchdog-action reset -boot menu=on -usb

Signed-off-by: Christian Nilsson <[email protected]>
@NiKiZe
Copy link
Owner Author

NiKiZe commented Jul 29, 2021

Have tried to create PoCs to generate PXE files for minimal livecd (amd64), feedback appreciated https://bugs.gentoo.org/494300#c4
One variant modifies upload step NiKiZe/releng@b2c1ffb
Another variant modifies ISO creation NiKiZe/catalyst@33c9405

NiKiZe added a commit to NiKiZe/releng that referenced this issue Aug 1, 2021
Extracts and generates needed files to perform iPXE boot from gentoo mirrors
Provided as PoC of what is needed for this to be vaild
If merged this would resolve NiKiZe/Gentoo-iPXE#2
Will be providing PoC for doing most of this in catalyst instead as well
Really want to have this in the same target as minimal livecd

To test result with qemu localy after build (replace your variables):
qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -nic user,model=virtio,\
tftp="${BUILD_SRCDIR_BASE}"/builds/default,\
bootfile=install-amd64-minimal-${TIMESTAMP}.ipxe \
-boot menu=on -usb

Signed-off-by: Christian Nilsson <[email protected]>
@NiKiZe
Copy link
Owner Author

NiKiZe commented Aug 10, 2021

robbat2:

mattst88: proj/releng.git scripts/copy_buildsync.sh creates latest-*.txt
plus catalyst-auto in some cases

dilfridge

Chewi: /etc/catalyst/catalystrc
on demeter it's
export MAKEOPTS="-j17"
export EMERGE_DEFAULT_OPTS=" --jobs=10 --load-average=32"
export FEATURES="${FEATURES} parallel-install"
(8 core machine)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

1 participant