nixos-rebuild: switch --flake fails on Out of Memory Silently

[CrystalSplitter]

Nixpkgs version

• Unstable (25.05)

Describe the bug

While running nixos-rebuild switch --flake ./nixos, on my 2GiB memory system, I noticed that it would "succeed" (no visible error), yet the system would remain the same. That is, nixos-rebuild would ultimately appear to do nothing. While the new environment would appear to be built through the *.config.system.build.toplevel, the new environment would not activate.

Looking around online, I found this Discourse forum thread https://discourse.nixos.org/t/sudo-nixos-rebuild-switch-does-nothing/9273, which perfectly reproduced my issue.

Running nixos-rebuild under strace shows that the thread is correct, I'm running out of memory and nixos-rebuild silently fails still with exit code 137. So the issue persists from 2020.

Steps to reproduce

Run nixos-rebuild switch --flake ./some-flake-here on a system with very little RAM and no swap. Perhaps try less than 0.5GiB in a VM?

Expected behaviour

nixos-rebuild exits with a clear error message when OoMing

Screenshots

No response

Relevant log output

$ strace nixos-rebuild switch --flake ./nixos/

-->8--

rt_sigreturn({mask=[]})                 = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0755, st_size=4096, ...}, 0) = 0
newfstatat(AT_FDCWD, "/nix/store/b1wvkjx96i3s7wblz38ya0zr8i93zbc5-coreutils-9.5/bin/rm", {st_mode=S_IFREG|0555, st_size=1443104, ...}, 0) = 0
newfstatat(AT_FDCWD, "/nix/store/b1wvkjx96i3s7wblz38ya0zr8i93zbc5-coreutils-9.5/bin/rm", {st_mode=S_IFREG|0555, st_size=1443104, ...}, 0) = 0
geteuid()                               = 1000
getegid()                               = 100
getuid()                                = 1000
getgid()                                = 100
access("/nix/store/b1wvkjx96i3s7wblz38ya0zr8i93zbc5-coreutils-9.5/bin/rm", X_OK) = 0
newfstatat(AT_FDCWD, "/nix/store/b1wvkjx96i3s7wblz38ya0zr8i93zbc5-coreutils-9.5/bin/rm", {st_mode=S_IFREG|0555, st_size=1443104, ...}, 0) = 0
geteuid()                               = 1000
getegid()                               = 100
getuid()                                = 1000
getgid()                                = 100
access("/nix/store/b1wvkjx96i3s7wblz38ya0zr8i93zbc5-coreutils-9.5/bin/rm", R_OK) = 0
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
rt_sigprocmask(SIG_BLOCK, [INT TERM CHLD], [], 8) = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fbf6e655a10) = 11974
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGINT, {sa_handler=0x447d20, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7fbf6e698620}, {sa_handler=0x46c8d0, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7fbf6e698620}, 8) = 0
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 11974
rt_sigaction(SIGINT, {sa_handler=0x46c8d0, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7fbf6e698620}, {sa_handler=0x447d20, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7fbf6e698620}, 8) = 0
ioctl(2, TIOCGWINSZ, {ws_row=53, ws_col=152, ws_xpixel=1368, ws_ypixel=954}) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=11974, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
wait4(-1, 0x7ffda2c5e7d0, WNOHANG, NULL) = -1 ECHILD (No child processes)
rt_sigreturn({mask=[]})                 = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
exit_group(137)                         = ?
+++ exited with 137 +++

Additional context

Ultimately, I do not think the flake.nix itself is relevant since it's quite minimal and this isn't about the flake itself, but on the error reporting. That being said, I can try to make a minimal example flake if asked. Additionally, I don't know if this only happens with flakes.

System metadata

$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.6.63, NixOS, 25.05 (Warbler), 25.05.20241204.6eac218`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.24.10`
 - channels(root): `"home-manager, nixos-24.11"`
 - nixpkgs: `/nix/store/ndfdhdw4i1cqijrl0cbnalvl37y4m0b4-source`

Notify maintainers

@Profpatsch

@thiagokokada

---

Note for maintainers: Please tag this issue in your pull request description. (i.e. Resolves #ISSUE.)

I assert that this issue is relevant for Nixpkgs

• I assert that this is a bug and not a support request.
• I assert that this is not a duplicate of an existing issue.
• I assert that I have read the NixOS Code of Conduct and agree to abide by it.

Is this issue important to you?

Add a 👍 reaction to issues you find important.

[thiagokokada]

Can you try nixos-rebuild-ng instead and see if you have the same issue? I assume you will get a better error message since nixos-rebuild-ng should always report the error code from each step.

[CrystalSplitter]

Can do!

Same command as last time (but with nixos-rebuild-ng)

[crystal@lamprey:~/dotfiles/nixos]$ nixos-rebuild-ng switch --flake ./
building the system configuration...
evaluating derivation 'path:/home/crystal/dotfiles/nixos#nixosConfigurations.lamprey.config.system.build.toplevel'Command '['nix', '--extra-experimental-features', 'nix-command flakes', 'build', '--print-out-paths', '.#nixosConfigurations.lamprey.config.system.build.toplevel', '--no-link']' died with <Signals.SIGKILL: 9>.

With --verbose

[crystal@lamprey:~/dotfiles/nixos]$ nixos-rebuild-ng switch --verbose --flake ./
building the system configuration...
debug: nixos_rebuild.process: calling run with args=['nix', '--extra-experimental-features', 'nix-command flakes', 'build', '--print-out-paths', '.#nixosConfigurations.lamprey.config.system.build.toplevel', '--no-link', '-v'], kwargs={'stdout': -1}, extra_env=None
evaluating derivation 'path:/home/crystal/dotfiles/nixos#nixosConfigurations.lamprey.config.system.build.toplevel'Traceback (most recent call last):
  File "/nix/store/j7j89bcny0panxiyjhli7w2282fmzvv3-nixos-rebuild-ng-0.0.0/bin/.nixos-rebuild-ng-wrapped", line 9, in <module>
    sys.exit(main())
             ^^^^^^
  File "/nix/store/j7j89bcny0panxiyjhli7w2282fmzvv3-nixos-rebuild-ng-0.0.0/lib/python3.12/site-packages/nixos_rebuild/__init__.py", line 408, in main
    execute(sys.argv)
  File "/nix/store/j7j89bcny0panxiyjhli7w2282fmzvv3-nixos-rebuild-ng-0.0.0/lib/python3.12/site-packages/nixos_rebuild/__init__.py", line 293, in execute
    path_to_config = nix.build_flake(
                     ^^^^^^^^^^^^^^^^
  File "/nix/store/j7j89bcny0panxiyjhli7w2282fmzvv3-nixos-rebuild-ng-0.0.0/lib/python3.12/site-packages/nixos_rebuild/nix.py", line 65, in build_flake
    r = run_wrapper(run_args, stdout=PIPE)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/j7j89bcny0panxiyjhli7w2282fmzvv3-nixos-rebuild-ng-0.0.0/lib/python3.12/site-packages/nixos_rebuild/process.py", line 130, in run_wrapper
    r = subprocess.run(
        ^^^^^^^^^^^^^^^
  File "/nix/store/zv1kaq7f1q20x62kbjv6pfjygw5jmwl6-python3-3.12.7/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['nix', '--extra-experimental-features', 'nix-command flakes', 'build', '--print-out-paths', '.#nixosConfigurations.lamprey.config.system.build.toplevel', '--no-link', '-v']' died with <Signals.SIGKILL: 9>.

Indeed, this is quite a bit clearer. Even just saying there's an error is an improvement.

[thiagokokada]

I will close this issue since nixos-rebuild-ng is now the default and it is fixed there.