--vmefi too small size of ESP partition / grub partition? / make size of ESP configurable (EFI)

[adrelanos]

  if [ -n "$VMEFI" ]; then
    parted -s "${TARGET}" 'mklabel gpt'
    parted -s "${TARGET}" 'mkpart ESP fat32 1MiB 101MiB'
    parted -s "${TARGET}" 'set 1 boot on'
    parted -s "${TARGET}" 'mkpart bios_grub 101MiB 102MiB'
    parted -s "${TARGET}" 'set 2 bios_grub on'
    parted -s "${TARGET}" 'mkpart primary ext4 102MiB 100%'

Could this lead to issues over time?

Too small? As per:
https://unix.stackexchange.com/questions/623304/size-of-efi-system-partition-esp

calamares uses 300M.

Roderick W. Smith (rEFIt boot manager developer) recommends:

, I recommend creating an ESP that's 550MiB in size.

That's why I would go with.

Quote https://www.rodsbooks.com/efi-bootloaders/principles.html

Although the EFI specification is mute on the subject of the ESP's size, most OSes make it fairly small—Macs ship with 200MiB ESPs, and the Windows 7 installer creates one of just 100MiB. (That value has been raised to a bit over 200MiB on newer versions of Windows.) Some users, however, have found that some EFIs have bugs that cause problems with FAT32 ESPs that are under 512MiB (537MB) in size. One very common problem is files that can't be read by the EFI. The Linux mkdosfs command defaults to using FAT16 for partitions of up to 520MiB (546MB). Therefore, adding a margin of safety to protect against MiB/MB confusion and rounding errors, I recommend creating an ESP that's at least 550MiB in size. If you must use a smaller ESP and if you encounter mysterious problems, try converting it to FAT16; most ESPs will work fine with this, and it may eliminate your problems. On the other hand, this may cause the Windows installer to fail should you need to install this OS.

https://www.rodsbooks.com/gdisk/advice.html#ESP_Sizing with more detailed reasoning:

550 MiB.

I'd like to use the grml-debootstrap created raw image as a base to create an ISO from it so it can be written to USB or DVD.
(Using grml-debootstrap, grml-live or other tools.) (#215)

Does that change the size requirements?

[mika]

IIRC the 300MB are also somewhere around what Windows uses for the EFI/ESP partition?

Its size depends on the actual use-case, not sure one does lots of stuff with it in a VM though, so there I would keep it smaller? But if you're creating raw images for usage on USB/DVD that might also be a matter to consider. Is there any actual disk image size change if we'd increase ESP size to 550MB?

[jkirk]

I think I also read all the given "ESP size recommendation" articles to decide on the "best" size for my ESP partition.

I usually choose 550 MiB on physical machines as well.

Things are different for VMs, especially when created via grml-debootstrap --vmefi`. Most of these systems are rather "short lived" or have a specific task. If I needed a system to test EFI tasks and/or if I needed more ESP size, I'd create a separate machine.

Creating a raw image to create an ISO should not require a larger ESP partition (is an ESP partition even needed for an ISO, @mika?). We should test that though and the default ESP size by grml-debootstrap should work OOTB for that task.

So yes, I support @mika's statement, to keep the the partition smaller, but make the ESP size configurable is probably a good idea.

JM2C.

[zeha]

Additional comments:

• The bios_grub partition could go away for VMs built with --vmefi. We're not having legacy boot support in such images anyway, right?
• With all the UKI/direct efiboot/... work ongoing upstream, maybe today is not the best time to make a lot of decisions in this area :)

[adrelanos]

• The bios_grub partition could go away for VMs built with --vmefi. We're not having legacy boot support in such images anyway, right?

Images for amd64 built with --vmefi are compatible with both, legacy BIOS and EFI booting. This was done in:

• Add support for legacy BIOS boot to efi VM images #216

Confirmed, tested.

[adrelanos]

Is there any actual disk image size change if we'd increase ESP size to 550MB?

Yes. Unfortunately so.

Changed the sizes for amd64. Related source code:

parted -s "${TARGET}" 'mklabel gpt'
parted -s "${TARGET}" 'mkpart ESP fat32 1MiB 551MiB' # Increased ESP size to 550MiB
parted -s "${TARGET}" 'set 1 boot on'
parted -s "${TARGET}" 'mkpart bios_grub 551MiB 552MiB' # Adjusted starting point of bios_grub
parted -s "${TARGET}" 'set 2 bios_grub on'
parted -s "${TARGET}" 'mkpart primary ext4 552MiB 100%' # Adjusted starting point of primary partition

I had to increase VMSIZE=2G (default as per git master) to VMSIZE=3G. (Otherwise the build would fail due to too small VM size with the new sizes.

I then compared the size of the resulting VM images:

• A) VMSIZE=3G with old partition sizes (old code as per git master)
  • efi-partition-old-size-with-vmsize-3g.img
  • du -sm: 1945 MB
  • du --apparent-size -sm: 3072 MB
• B) VMSIZE=3G with new partition sizes (new code as per this comment)
  • efi-partition-new-size-with-vmsize-3g.img
  • du -sm: 1826 MB
  • du --apparent-size -sm: 3072 MB

zerofree didn't help. zerofree cannot be used on FAT32 (ESP).

[mika]

Is there any actual disk image size change if we'd increase ESP size to 550MB?

Yes. Unfortunately so.

Ok :-/

Changed the sizes for amd64. Related source code:

parted -s "${TARGET}" 'mklabel gpt'
parted -s "${TARGET}" 'mkpart ESP fat32 1MiB 551MiB' # Increased ESP size to 550MiB
parted -s "${TARGET}" 'set 1 boot on'
parted -s "${TARGET}" 'mkpart bios_grub 551MiB 552MiB' # Adjusted starting point of bios_grub
parted -s "${TARGET}" 'set 2 bios_grub on'
parted -s "${TARGET}" 'mkpart primary ext4 552MiB 100%' # Adjusted starting point of primary partition

I had to increase VMSIZE=2G (default as per git master) to VMSIZE=3G. (Otherwise the build would fail due to too small VM size with the new sizes.

I then compared the size of the resulting VM images:

* A) `VMSIZE=3G` with old partition sizes (old code as per git master)
  
  * `efi-partition-old-size-with-vmsize-3g.img`
  * `du -sm`: `1945` MB
  * `du --apparent-size -sm`: `3072` MB

* B) `VMSIZE=3G` with new partition sizes (new code as per this comment)
  
  * `efi-partition-new-size-with-vmsize-3g.img`
  * `du -sm`: `1826` MB
  * `du --apparent-size -sm`: `3072` MB

Hm, I'm not sure I understand: in git master we have:

parted -s "${TARGET}" 'mkpart ESP fat32 1MiB 101MiB'

And this resulted in a 1945 MB image for you, though with:

parted -s "${TARGET}" 'mkpart ESP fat32 1MiB 551MiB' # Increased ESP size to 550MiB

it shrinks to 1826 MB, or am I missing something? 🤔

[adrelanos]

That seems confusing indeed. To make sure I didn't make a mistake, just now I re-created this experiment.

du -sm *

1834	efi-partition-new-size-with-vmsize-3g.img
1774	efi-partition-old-size-with-vmsize-2g.img
1905	efi-partition-old-size-with-vmsize-3g.img

du --apparent-size -sm *

3072	efi-partition-new-size-with-vmsize-3g.img
2048	efi-partition-old-size-with-vmsize-2g.img
3072	efi-partition-old-size-with-vmsize-3g.img

Indeed, with bigger EFI partition sizes the image shrinks. I have no explanation for this.

Most important point is that without VMSIZE=3G it won't build with the bigger EFI partition sizes.

Is a change of the defaults VMSIZE=3G and the increased EFI partition size an option? The increase would be from 1774 MB to 1834 MB. Not that much. Though the "--apparent-size" would increase from 2 GB to 3 GB.

If that is not an option, would a PR to make to configurable would be acceptable?

---

If a PR, here are some thoughts on the implementation details.

AMD64 --vmefi:

parted -s "${TARGET}" 'mkpart ESP fat32 1MiB 101MiB'
parted -s "${TARGET}" 'mkpart bios_grub 101MiB 102MiB'
parted -s "${TARGET}" 'mkpart primary ext4 102MiB 100%'

Should,

• A) each number 1, 101, 102 become a variable? Documentation could showcase one example using 550MIB on how to set these 3 variables. Or,
• B) Invent a new variable esp_partition_size could be the other numbers based on calculations.

?

• A) Would be easier in the source code but harder for users.
• B) Would be easier for users but more complex source code.

A) would be preferred by me because I don't think many users will play with this.

Also these numbers are different for ARM64 and legacy BIOS AMD64 builds. Calculating this seems complex. How much configurability should be possible? Somebody providing their own parted script?

Just up to the one doing the PR?

[zeha]

Indeed, with bigger EFI partition sizes the image shrinks. I have no explanation for this.

I would imagine FAT doesn't touch unused blocks, so the unused space on the ESP ends up truly free. ext4 probably writes some block group data ...

[mika]

@adrelanos

Most important point is that without VMSIZE=3G it won't build with the bigger EFI partition sizes.

Is a change of the defaults VMSIZE=3G and the increased EFI partition size an option? The increase would be from 1774 MB to 1834 MB. Not that much. Though the "--apparent-size" would increase from 2 GB to 3 GB.

I think changing default to VMSIZE=3G is worth a try?

If that is not an option, would a PR to make to configurable would be acceptable?

If a PR, here are some thoughts on the implementation details.

AMD64 --vmefi:

parted -s "${TARGET}" 'mkpart ESP fat32 1MiB 101MiB'
parted -s "${TARGET}" 'mkpart bios_grub 101MiB 102MiB'
parted -s "${TARGET}" 'mkpart primary ext4 102MiB 100%'

Should,

* A) each number 1, 101, 102 become a variable? Documentation could showcase one example using 550MIB on how to set these  3 variables. Or,

* B) Invent a new variable `esp_partition_size` could be the other numbers based on calculations.

?

* A) Would be easier in the source code but harder for users.

* B) Would be easier for users but more complex source code.

A) would be preferred by me because I don't think many users will play with this.

Also these numbers are different for ARM64 and legacy BIOS AMD64 builds. Calculating this seems complex. How much configurability should be possible? Somebody providing their own parted script?

Just up to the one doing the PR?

I also feeel that option A is the way to go here, assuming we don't want to hardcode anything specific?