Filesystem optimizations

[matthewjmarangoni]

This PR adds filesystem optimizations which on the surface exchange an ~11% response time, ~20% Chromium startup time, and ~67% filesystem build time for an image size reduction of ~48% and initial boot time reduction of ~39%. Although this PR does not depend on another PR, when combined with the other pending optimizations the benefits compound.

The advantages continue with effects through the chain. The image size reduction is generally interesting and brings additional improvements to deployment times (e.g. publish, propagation, build). Initial boot time improvements are present but might yield the most benefit outside general use cases. There are pulls on time to first response an order of magnitude larger than the boot time reduction herein.

Further testing is needed. This PR remains separate from other optimizations not only from the focus of the content and potential for unsuitability but also due to the higher potential for negative impact. If inapplicable for the default operating case it could remain an interesting option for special cases.

Some future benefits might be inherited automatically or made available as EROFS is further developed. The integration of zstandard compression is one such item that stands out.

Anecdotal testing, suggested to use EROFS 1.7:

• headless:
  • 4s (~60%) slower build time
  • standard: 18.765ms average, 15.15ms min, 21.68ms max, 1.5GB image size
  • modified: 20.518ms average, 16.71ms min, 23.80ms max, 763MB image size
  • delta: +1.75ms average, +1.56ms min, +2.12ms max, 773MB image size reduction, ~300ms faster first boot time
  • delta: ~10% higher response time average (slower), ~11% higher response time min (slower), ~10% higher response max (slower), ~50% lower space usage, ~39% lower first boot time (faster)
• headful:
  • 7s (~67%) slower filesystem build time, 1s (~20%) slower Chromium start up from launch to debugging port availability
  • 745ms (~60%) faster first boot time
  • 1.3GB (~48%) image size reduction

---

TL;DR

Switched the unikernel filesystem to EROFS with LZ4 compression, trading a minor runtime performance cost for a ~48% reduction in image size and ~39% faster initial boot times.

Why we made these changes

To significantly reduce the final image size. Smaller images lead to faster deployment times (publishing, propagation, builds) and lower storage costs. While this optimization introduces a slight increase in response times and build times, the substantial benefits to the deployment lifecycle and boot performance make it a worthwhile trade-off to explore.

What changed?

• Updated the build-unikernel.sh scripts for both chromium-headful and chromium-headless.
• The mkfs.erofs command is now used to build the initrd filesystem with the following key changes:
  • Enabled LZ4 compression (-z lz4).
  • Added support for fragments (-E fragments).
  • Set a cluster size of 131072 bytes.
  • Disabled extended attributes (-x 0).

Validation

Anecdotal testing shows:

• Image Size: Reduced by ~48% (~1.3GB) for headful and ~50% (~773MB) for headless.
• Boot Time: Initial boot is ~39% faster.
• Performance Trade-offs:
  • ~11% slower average response time.
  • ~20% slower Chromium startup time.
  • ~67% slower filesystem build time.

Further testing is needed, as these changes have a higher potential for negative performance impact in the default operating case.

^{Description generated by Mesa. Update settings}

[mesa-dot-dev]

Performed full review of 330685e...c02e552

Tip

⚡ Quick Actions

This review was generated by Mesa.

Actions:

• Configure your agents

Slash Commands:

• /review - Request a full code review
• /review latest - Review only changes since the last review
• /describe - Generate PR description. This will update the PR body or issue comment depending on your configuration
• /help - Get help with Mesa commands and configuration options

^{2 files reviewed | 3 comments | Review on Mesa | Edit Reviewer Settings}

[mesa-dot-dev]

The -x 0 parameter being added here should be documented or explained. The -x flag in mkfs.erofs typically controls xattr features, and setting it to 0 disables extended attribute support. This could impact applications that rely on extended attributes for security contexts, capabilities, or other metadata. Consider documenting why this is being disabled and whether it's intentional for the optimization goals.
_{Agent: 🤖 General}

[mesa-dot-dev]

Inconsistency detected: This command doesn't use sudo while the equivalent command in images/chromium-headful/build-unikernel.sh does use sudo. Both commands create the same initrd file and use identical parameters. This inconsistency could lead to permission issues or different behavior between headful and headless builds. Consider standardizing the approach - either both should use sudo or neither should, depending on the execution environment and file permissions needed.
_{Agent: 🤖 General}

[mesa-dot-dev]

The compression block size -C 131072 (128KB) is quite large and may not be optimal for all use cases. While this can improve compression ratios for large files, it can hurt performance for small file access patterns common in web applications. The PR mentions this is experimental and needs further testing - consider adding a comment explaining the rationale for this specific block size choice and making it configurable via an environment variable for easier testing of different values.
_{Agent: 🤖 General}

[mesa-dot-dev]

Performed full review of 330685e...c02e552

Tip

⚡ Quick Actions

This review was generated by Mesa.

Actions:

• Configure your agents

Slash Commands:

• /review - Request a full code review
• /review latest - Review only changes since the last review
• /describe - Generate PR description. This will update the PR body or issue comment depending on your configuration
• /help - Get help with Mesa commands and configuration options

^{2 files reviewed | 3 comments | Review on Mesa | Edit Reviewer Settings}

[mesa-dot-dev]

The compression block size of 131072 (128KB) is quite large and may increase memory usage during filesystem operations. Consider documenting the minimum EROFS-utils version required (1.6+) for the -C and -z flags, or add version validation to prevent build failures on older systems.
_{Agent: 🤖 General}

[mesa-dot-dev]

Inconsistency detected: The headful version uses 'sudo' for mkfs.erofs but this headless version doesn't. This could lead to permission issues depending on the build environment. Consider aligning the privilege requirements between both scripts.
_{Agent: 🤖 General}

[mesa-dot-dev]

Consider adding error handling to validate that the EROFS-utils version supports the newer flags (-C, -z) before executing. The build will fail silently on systems with EROFS-utils < 1.6, making debugging difficult.
_{Agent: 🤖 General}

[mesa-dot-dev]

Performed full review of 330685e...c02e552

Tip

⚡ Quick Actions

This review was generated by Mesa.

Actions:

• Configure your agents

Slash Commands:

• /review - Request a full code review
• /review latest - Review only changes since the last review
• /describe - Generate PR description. This will update the PR body or issue comment depending on your configuration
• /help - Get help with Mesa commands and configuration options

^{2 files reviewed | 3 comments | Review on Mesa | Edit Reviewer Settings}

[mesa-dot-dev]

The parameter ordering could be improved for readability. Consider grouping related parameters together, e.g., placing compression-related options (-C, -z) adjacent to each other: sudo mkfs.erofs --all-root -d2 -E noinline_data,fragments -b 4096 -C 131072 -z lz4 -x 0 initrd ./.rootfs
_{Agent: 🤖 General}

[mesa-dot-dev]

Consider adding error handling for the mkfs.erofs command. Since this is a critical step in the build process and the new parameters could potentially fail with older EROFS versions, adding || { echo "EROFS filesystem creation failed"; exit 1; } would provide better error reporting.
_{Agent: 🤖 General}

[mesa-dot-dev]

Inconsistency: The headful version uses sudo for mkfs.erofs while this headless version doesn't. This could lead to different permission behaviors or failures depending on the execution environment. Consider either adding sudo here or documenting why it's not needed for the headless build.
_{Agent: 🤖 General}