feat/lower squashfs size #211

blaggacao · 2024-05-10T13:26:53Z

refactor: bash+jq instead of python closure
chore: anticipate expected delay in tests
feat: free caches and log free mem before kexec
remove some extra weight

Context

Some edge environments have less than the required 1.5 GB RAM
One can disable file system support, however that alone is not enough
The base system also needs to be trimmed down
This PR is contributing to trimming the base system down, regardless of the fs support

Results

The potential savings where analyzed on the likes of:

nix-tree /nix/store/0szqjjq5r98063cvmnhq7yrwb6dmxsvy-nixos-system-nixos-installer-23.11pre-git

This change reduces the overall size of the unpacked squashfs store in the order of the sum of the savings comments
I'm currently testing to go further (removing pearl in addition to fs support) and have arrived at around ~500MB unpacked vs initial some ~1GB unpacked (values are just a ballpark number and probably not yet the end)

Unstable Patch Required

From 0c34c3c822d82c60f06783e834a9c602223ee6a8 Mon Sep 17 00:00:00 2001
From: David <[email protected]>
Date: Fri, 10 May 2024 16:57:00 +0200
Subject: [PATCH] fix: systemd build flag combinations

---
 pkgs/os-specific/linux/systemd/default.nix | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/pkgs/os-specific/linux/systemd/default.nix b/pkgs/os-specific/linux/systemd/default.nix
index 9cdc5dcd9d44..8f469714db92 100644
--- a/pkgs/os-specific/linux/systemd/default.nix
+++ b/pkgs/os-specific/linux/systemd/default.nix
@@ -562,7 +562,7 @@ stdenv.mkDerivation (finalAttrs: {
     (lib.mesonEnable "zlib" withCompression)
 
     # NSS
-    (lib.mesonEnable "nss-mymachines" withNss)
+    (lib.mesonEnable "nss-mymachines" (withNss && withMachined))
     (lib.mesonEnable "nss-resolve" withNss)
     (lib.mesonBool "nss-myhostname" withNss)
     (lib.mesonBool "nss-systemd" withNss)
@@ -574,7 +574,7 @@ stdenv.mkDerivation (finalAttrs: {
 
     # FIDO2
     (lib.mesonEnable "libfido2" withFido2)
-    (lib.mesonEnable "openssl" withFido2)
+    (lib.mesonEnable "openssl" (withHomed || withFido2 || withSysupdate))
 
     # Password Quality
     (lib.mesonEnable "pwquality" withPasswordQuality)
-- 
2.42.0

blaggacao · 2024-05-14T14:39:30Z

Green depends on: NixOS/nixpkgs#311675

Mic92 · 2024-05-19T10:44:52Z

nix/kexec-installer/kexec-run.sh

@@ -74,6 +74,9 @@ if ! "$SCRIPT_DIR/kexec" --load "$SCRIPT_DIR/bzImage" \
  exit 1
 fi

+sync; echo 3 > /proc/sys/vm/drop_caches
+echo "current available memory: $(free -h | awk '/^Mem/ {print $7}')"


Doesn't the linux kernel do this automatically anyway if it needs memory.

I could very well imagine that yes, although I didn't verify and don't intent to stake that claim either.

The main reason I put this here is for reporting so that (especially during tests) we can see how much effective free memory was available prior to switching the kernel.

Not sure if it is utterly useful otherwise.

Ok. But you are already printing the available memory, which will automatically substract any memory claimed by page cache or dirty pages.
So no need to flush the caches, which might even contain the initrd that we are about to kexec into.

Here we could even print a warning if we are below X MB available memory.

Here we could even print a warning if we are below X MB available memory.

Yeah, I had that idea, too, today: outright refuse to proceed because it's potentially not recoverable, if you have no IPMI access.

Ok. But you are already printing the available memory, which will automatically substract any memory claimed by page cache or dirty pages.
So no need to flush the caches, which might even contain the initrd that we are about to kexec into.

Iirc, I did some A/B testing and saw a small ~30MB difference, but yeah, I agree that it shouldn't be necessary.

I did notice, however, on a slightly tangential note, that at the RAM-limit, kexec --load kept working while then kexec -e failed. But I guess that's just due to what the stage 1 or squashfs ultimately command into RAM.

nix/installer.nix

Mic92 · 2024-05-19T11:35:31Z

nix/kexec-installer/module.nix

-  restore-network = pkgs.writers.writePython3 "restore-network" { flakeIgnore = [ "E501" ]; }
-    ./restore_routes.py;
+
+  restore-network = pkgs.writers.writeBash "restore-network" ./restore_routes.sh;


What's the saving removing python out of interest?

I don't remember exactly. It was relatively significant though. I may even think in the ballpark of 150MB or so. It was, indeed, the lowest hanging fruit. I could have gone with python minimal, but since I was trying to really size this down as much as possible, I thought I'd save those additional dozens of MB that python minimal would have left us with, as well.

However, the effect on the RAM was rather small (maybe 30-60MB or so?), ostensible due to a somewhat efficient caching and on-demand decompression of the squashfs, while having a (much!) smaller impact while compressed.

blaggacao · 2024-05-19T12:30:31Z

Closures

Baseline:

      util-linux = prev.util-linux.override {
        nlsSupport = false;
        ncursesSupport = false;
        systemdSupport = false;
        translateManpages = false;
      };

initrd 635.4 MiB (635.4 MiB)

w translateManpages

      util-linux = prev.util-linux.override {
        nlsSupport = false;
        ncursesSupport = false;
        systemdSupport = false;
        # translateManpages = false;
      };

initrd 635.4 MiB (635.4 MiB)

w systemdSupport

      util-linux = prev.util-linux.override {
        nlsSupport = false;
        ncursesSupport = false;
        # systemdSupport = false;
        # translateManpages = false;
      };

initrd 649.33 MiB (649.33 MiB)

Interestingly, bringing back systemd support, despite using systemd seems to have negative effect for other reasons than the pure dependency.

w ncursesSupport

      util-linux = prev.util-linux.override {
        nlsSupport = false;
        # ncursesSupport = false;
        # systemdSupport = false;
        # translateManpages = false;
      };

initrd 639.23 MiB (639.23 MiB)

original

      # util-linux = prev.util-linux.override {
        # nlsSupport = false;
        # ncursesSupport = false;
        # systemdSupport = false;
        # translateManpages = false;
      # };

initrd 651.07 MiB (651.07 MiB)

blaggacao · 2024-05-19T12:57:55Z

NixOS/nixpkgs@e5b250b is still on staging and thus CI isn't yet green. It will probably take another short while before we can pull this in with the latest unstable.

nix/noninteractive.nix

nix/no-grub.nix

nix/netboot-installer/module.nix

Mic92 · 2024-05-19T14:43:06Z

We are now down to 1GB: https://github.com/nix-community/nixos-images/pull/218/files

nix/kexec-installer/test.nix

Mic92 · 2024-05-19T16:29:21Z

nix/installer.nix

+      # save ~12MB by not bundling manpages
+      coreutils-full = prev.coreutils;


Also question here: Do we actually care about this? We are not shipping manpages afaik and they are hopefully in a separate output?

they are hopefully in a separate output?

Unfortunately not, at least in nixos-23.11:

postInstall = optionalString (isCross && !minimal) '' rm $out/share/man/man1/* cp ${buildPackages.coreutils-full}/share/man/man1/* $out/share/man/man1 '' # du: 8.7 M locale + 0.4 M man pages + optionalString minimal '' rm -r "$out/share" '';

minimal is false on -full

Mic92 · 2024-05-19T16:33:45Z

nix/kexec-installer/restore_routes.sh

@@ -0,0 +1,121 @@
+#!/usr/bin/env bash


cc @Lassulus for review of this.

nix/kexec-installer/test.nix

blaggacao force-pushed the feat/lower-squashfs-size branch 2 times, most recently from e7b659f to 9bd468e Compare May 10, 2024 16:31

blaggacao mentioned this pull request May 19, 2024

feat/lower squashfs size unpearl #212

Closed

blaggacao force-pushed the feat/lower-squashfs-size branch from cb56c9d to ef9a889 Compare May 19, 2024 09:26

Mic92 reviewed May 19, 2024

View reviewed changes

nix/installer.nix Show resolved Hide resolved

Mic92 reviewed May 19, 2024

View reviewed changes

nix/installer.nix Show resolved Hide resolved

Mic92 reviewed May 19, 2024

View reviewed changes

nix/installer.nix Outdated Show resolved Hide resolved

Mic92 reviewed May 19, 2024

View reviewed changes

nix/noninteractive.nix Outdated Show resolved Hide resolved

Mic92 reviewed May 19, 2024

View reviewed changes

nix/noninteractive.nix Outdated Show resolved Hide resolved

Mic92 reviewed May 19, 2024

View reviewed changes

nix/noninteractive.nix Outdated Show resolved Hide resolved

Mic92 reviewed May 19, 2024

View reviewed changes

nix/no-grub.nix Outdated Show resolved Hide resolved

Mic92 reviewed May 19, 2024

View reviewed changes

nix/netboot-installer/module.nix Outdated Show resolved Hide resolved

blaggacao force-pushed the feat/lower-squashfs-size branch from 0541cd6 to df11b55 Compare May 19, 2024 14:38

blaggacao added 7 commits May 19, 2024 18:00

refactor: bash+jq instead of python closure

2881477

chore: anticipate expected delay in tests

deff430

feat: free caches and log free mem before kexec

f274f19

remove some extra weight

a087c0d

chore: iterate systemd slim

51f092e

chore: make util-linux modifications more concise

6275988

refactor: adapt to cherry pick

eeb2377

blaggacao force-pushed the feat/lower-squashfs-size branch from c54ae6f to eeb2377 Compare May 19, 2024 16:00

Mic92 reviewed May 19, 2024

View reviewed changes

nix/kexec-installer/test.nix Show resolved Hide resolved

Mic92 reviewed May 19, 2024

View reviewed changes

nix/kexec-installer/restore_routes.sh

@@ -0,0 +1,121 @@

#!/usr/bin/env bash

Copy link

Member

Mic92 May 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @Lassulus for review of this.

blaggacao added 2 commits May 19, 2024 19:22

fix: docstrings

a8032e0

fix: cleanup after rebase

b9eec55

Mic92 reviewed May 19, 2024

View reviewed changes

nix/kexec-installer/test.nix Show resolved Hide resolved

nix/kexec-installer/test.nix: comment why we wait 6s

2251e84

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat/lower squashfs size #211

feat/lower squashfs size #211

blaggacao commented May 10, 2024 •

edited

Loading

blaggacao commented May 14, 2024

Mic92 May 19, 2024

blaggacao May 19, 2024 •

edited

Loading

Mic92 May 19, 2024

Mic92 May 19, 2024

blaggacao May 19, 2024 •

edited

Loading

blaggacao May 19, 2024

Mic92 May 19, 2024

blaggacao May 19, 2024

blaggacao May 19, 2024 •

edited

Loading

blaggacao commented May 19, 2024 •

edited

Loading

blaggacao commented May 19, 2024

Mic92 commented May 19, 2024

Mic92 May 19, 2024

blaggacao May 19, 2024

Mic92 May 19, 2024

		# save ~12MB by not bundling manpages
		coreutils-full = prev.coreutils;

feat/lower squashfs size #211

Are you sure you want to change the base?

feat/lower squashfs size #211

Conversation

blaggacao commented May 10, 2024 • edited Loading

Context

Results

blaggacao commented May 14, 2024

Choose a reason for hiding this comment

blaggacao May 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blaggacao May 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blaggacao May 19, 2024 • edited Loading

Choose a reason for hiding this comment

blaggacao commented May 19, 2024 • edited Loading

Closures

blaggacao commented May 19, 2024

Mic92 commented May 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

blaggacao commented May 10, 2024 •

edited

Loading

blaggacao May 19, 2024 •

edited

Loading

blaggacao May 19, 2024 •

edited

Loading

blaggacao May 19, 2024 •

edited

Loading

blaggacao commented May 19, 2024 •

edited

Loading