[daniel-noland]

Stack (NIC offload; review and merge in order): #2030 (1) → #2031 (2) → #2032 (3) → this PR (3a); #2033 (4) is parked.

Production gives the gateway far more 1 GiB hugepages than the default rings need (vlab does not). Init now treats an allotment as a budget and sizes the receive rings to it.

• --hugepages-gib N is a budget of N 1 GiB pages per NIC NUMA node.
  • The pages must be available: there is no fallback to 2 MiB pages.
  • The ports and workers must fit, or init fails with what they need.
  • Workers are taken as given.
• Interfaces without /rxd=N get the deepest of 4096, 2048 and 1024 receive descriptors per queue whose pools leave RESERVE_MIB (512 MiB) of the budget free. The choice is made per node, logged, and passed to the dataplane in the launch configuration. Nothing deeper than 4096 is chosen.
• Without --hugepages-gib, 4096 or 2048 only if 1 GiB pages that are already free hold that with the reserve; init does not grow the pools for a deeper ring. Otherwise 1024, with the requirement computed (and the pools grown) as before.
• An explicit /rxd=N is never changed.
• The dataplane publishes its DPDK heap per NIC node (dpdk_heap_bytes{node,state}, dpdk_heap_largest_free_bytes{node}, and a startup log line). What the pools, ACL tables and drivers hold of the hugepages was not visible before: EAL maps all of --numa-mem up front and runs without telemetry.

Reserve, measured on a BlueField-3 with two ports and NIC offload on:

Workers / rxd	Predicted pools	Heap allocated, no policy	+ 16k ACL denies and 2k rate limits, all offloaded
4 / 1024	334 MiB	429 MiB (+95)	+92 MiB
16 / 4096	2,525 MiB	2,668 MiB (+143)	+58 MiB (4k rules)
32 / 4096	5,049 MiB	5,245 MiB (+196)	+92 MiB, peak +99 MiB while every rule changed

Beyond the pools that is about 80 MiB plus 3.6 MiB per worker, and at most about 99 MiB for the largest policy tried. The worst case, about 295 MiB, fits within the existing 256 MiB overhead plus the 512 MiB reserve.

Tested on the BF-3:

Case	Result
4 GiB allotment, 4 workers	4096 descriptors (888 MiB of 4096 MiB, 512 MiB to spare)
1 GiB allotment, 32 workers	refused at startup: needs 2929 MiB at 1024
64 GiB allotment, only 4 pages on the host	refused at startup, no 2 MiB fallback
No allotment, 4 workers, 4 free 1 GiB pages	4096
No allotment, 32 workers	1024 on 3 GiB, as before

Unit tests cover choosing within an allotment, a too-small or unavailable allotment, and deeper rings from free pages only.

Two flaky tests on main, fixed here so CI is quiet:

• power::tests::an_idle_wait_sleeps_until_its_deadline is ignored for now, with a note in its doc comment. On AMD, DPDK's rte_power_monitor hands MWAITX the low 32 bits of the absolute TSC deadline, but MWAITX's timer takes the time to wait (DPDK 1213a8895b, still so upstream), so an idle wait ends at the next interrupt: 2 µs to 1 ms for a 20 µs wait on a Zen 4. Passing MWAITX the time to wait fixed it on a Threadripper 7000 (family 0x19 model 0x18), but on the Ryzen 7000 CI runner (model 0x61) every wait then returned in about 450 ns, so no fix is in this PR. When run with --ignored, a failure now reports the CPU, the sleeper and every wait's duration.
• routing test_cli_handling read the last-applied FRR config before the fake FRR agent had answered. It now waits for it (40 of 40 runs pass).

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

• Source mode: Auto
• Repository: Unavailable
• Branch: Unavailable
• File: No configuration file from this repository applied
• Commit: Unavailable
• Layers: Organization UI
• Review profile: CHILL
• Plan: Essentials
• Run ID: 0b53eaf5-8e09-4b39-bce5-6250f0b45c58

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

CodeRabbit Verify

• Run CodeRabbit Verify to test the changes in this pull request.

[codecov]

⚠️ JUnit XML file not found

The CLI was unable to find any JUnit XML files to upload.
For more help, visit our troubleshooting guide.

[mvachhar]

Note

🤖 AI-drafted (Claude) on behalf of @mvachhar, reviewed by him before posting.

Approving: no blockers, and deeper rings sized to the hugepages are a good change. Verified:

• Without --hugepages-gib, the old path is unchanged, and the deeper-ring path can't introduce a failure.
• The cgroup allowance is honored on every path.
• One shared formula: pool sizing lives in args::dpdk_memory, shared by init and the dataplane, so they can't drift.
• Ports that can't take 4096 descriptors are clamped by the dataplane; init only over-reserves.
• Heap metrics are cheap (once a second), and the routing test's wait is bounded.

Should fix (not blocking):

1. --hugepages-gib means different things to fabric and the dataplane. Fabric's gateway controller (pkg/ctrl/gateway_ctrl.go:557-581) passes gatewayHugePages1G = N both as the pod's hugepages-1Gi limit (N GiB total) and as --hugepages-gib N. The dataplane treats N as a budget per NIC NUMA node. On a gateway with NICs on two nodes, init plans 2N GiB against a cgroup limit of N and fails. The mismatch predates this PR, but now that the allotment is a strict budget with no 2 MiB fallback, it's worth settling: either a total here, or fabric multiplying by NIC nodes. @Frostman, which do you prefer? Please also update the stale help and field docs (args/src/lib.rs:1810-1813, ~:643): they still say it "overrides the amount init computes", and mention neither "budget" nor "1 GiB pages only".

2. The doc comment on the ignored power test is wrong (dpdk/src/power/tests.rs, and the PR description too). It says the EAL hands MWAITX the low 32 bits of the absolute deadline and that the relative-time fix isn't in. It is in: amd-mwaitx-relative-timeout.patch is applied in nix/pkgs/dpdk/default.nix:49-50, on main since 16a6967. With it applied, the remaining issue is waits returning in about 450 ns on the Ryzen 7000 runner. Please fix the comment so it doesn't send the next reader after an already-fixed bug.

3. Tests: reserve_for itself isn't tested. That means:

   • the precedence (allotment → deeper rings in free pages → the old path);
   • only ports without /rxd being changed;
   • multi-node allotments with a different depth per node;
   • the assumption in deepest_fitting that RX_DESCRIPTOR_CHOICES is sorted with RX_DESCRIPTORS last.

   A property test (development/code/property-testing.md) would cover it well: for any need and budget, the chosen ring plus the reserve fits, and nothing deeper does.

4. SocketId::heap (dpdk/src/socket.rs:91-105) is a safe pub(crate) fn that calls rte_malloc_get_socket_stats, which reads EAL's memory config. That's undefined behavior before EAL init. The SAFETY comment relies on a manager existing, but Dev::heaps calls it without one. Please take &Manager/&Eal, or tie it to the 'eal borrow.

5. The chosen depth goes back to the dataplane by re-parsing PCI addresses (name_rx_descriptors, init/src/main.rs:1110), and a parse failure is skipped silently. Carrying the depth on the resolved device would keep the invariant in a type. A miss is safe today, since the dataplane falls back to 1024, which init over-reserved for.

Notes:

• Misleading log: within_allotment logs a chosen depth for a node even when every port there set /rxd.
• "Per node" only applies with an allotment. Without one, a single depth is chosen for all nodes; the PR text says per node.
• Page premapping: the deeper-ring path premaps pools + 512 MiB in free 1 GiB pages, more than before. Fine on gateway nodes, but worth a release note.
• Doc drift: the RESERVE_MIB doc says "~95 MiB … ~310 MiB"; the PR body says "~80 MiB … ~295 MiB".
• CI: the docs step of check/debug is failing.
• Slop (~30 lines):
  • the test-only place shim in init/src/hugepages.rs (~14 lines);
  • the trivial Manager::heap wrapper (~8 lines);
  • measurement prose duplicated between RESERVE_MIB and the PR body.