Repository navigation
Bump dataplane version - #2039
Closed
hedgehog-dataplane-update[bot] wants to merge 43 commits into
Closed
hedgehog-dataplane-update[bot] wants to merge 43 commits into
hedgehog-dataplane-update[bot] wants to merge 43 commits into
Conversation
SyncObserve, SyncCreate, SyncUpdate, SyncRemove, SyncReconcile and SyncOp mirror the asynchronous traits and their contracts, for external systems a thread drives with blocking calls, such as a NIC's flow rules programmed through a queue the thread waits on. Acting methods take &mut self: a synchronous reconciler usually owns the handles of what it created, and acting changes them. The asynchronous traits take &self so their futures can be shared and sent. observe takes &'a self, so an observation may borrow those handles. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
A rule whose traffic no earlier rule can claim keeps its meaning in any order, which is what a backend without first-match order (a NIC's hash tables) needs to know before it takes a rule. - Overlaps, per field kind: exact (equal), prefix (agree on the shorter length), mask (agree where both care), range (intersect). Property tests: a key both rules accept implies overlap, and an overlap has a witness key. - The derive generates overlaps() on every rule type: every field overlaps. - Generated items allow missing_docs, so a public key can derive. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
An offload lowers the same typed rules the software classifier matches with, rather than re-deriving them from config: PeeringKey and its rule are public, PeeringRules::v4()/v6() return the rules in first-match order with the configured rule each came from, and RateLimits::classifier() reaches them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
So that a NIC offload can lower the ACLs, and its tests can check the NIC against the filter: - AclKey and its derived AclKeyRule are public, and AclFilterContext::v4()/v6() return the rules in first-match order with what each decides (LookupResult: action, scope, whether it logs). - AclFilterContext::direct_match classifies a packet with no flow state: the first rule matching it directly, or why the filter drops a packet it cannot classify. A direct match is final in the filter, whatever the flow state; default_action gives the rest. - With the new `reference` feature, AclFilterContext::reference builds a context with the linear-scan classifier, without the EAL, for other crates' tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Codex <noreply@openai.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
FlowFilterContext::destinations lists the first stage's destinations, in match order: from a source VPC, a peer's public prefix, whether only replies on a masquerade flow reach it (gated), and whether it takes only some protocols or ports (port forwarding). The first that matches a packet is the VPC it goes to. A rule's destination prefix only stands for a destination VPC as far as these say: NIC offload narrows its rules with them. The reference feature builds a context with the linear-scan classifier, so crates can check against the flow filter without an EAL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
A new crate lowers the published rate limits into NIC hardware as a cascade in front of the software limiter: the NIC may only drop what software would drop too, and never changes a packet, so the pipeline stays authoritative and unchanged, and a NIC that takes nothing is always correct. - program compiles RateLimits<PeeringRules> into a backend-neutral Program (meters, rules, refusals) from the typed rules the software classifier keeps. It knows no NIC. Refused, and left to software: rules that overlap an earlier rule charging another meter (hardware tables keep no order), TCP flags, port ranges and partial protocol masks, ports without TCP or UDP, any-destination rules (the destination prefix stands in for the destination VPC), and meters that drop nothing. Of a meter chain the NIC enforces the first component that drops red, bytes before packets: conservative, since the chain only ever makes a color worse. - mlx5 lowers a Program onto one port with template rules: a root matching the IPv4 VXLAN underlay, one table per match shape (inner family, which prefixes, ports), each rule metering into a per-rule METER_MARK and jumping to its meter's color group, which counts and sends green and yellow to the fate (RSS over the data queues) and drops red. It follows the rate-limit reader, rebuilds on change, fails open, logs counts, and exports rate_limit_nic_packets_total. - probe_args and port_resources tell the driver what a device needs at probe (dv_flow_en=2, and a service_core the process may run on: mlx5 otherwise pins its counter thread to the main lcore's id used as a CPU number, and in a confined process port configuration fails and the PMD then segfaults) and what a port reserves before start. Prototype: only with DATAPLANE_NIC_OFFLOAD=1, mlx5 only. Rates count outer bytes; reconfiguration is teardown then rebuild, briefly unlimited in hardware. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
The DPDK driver hands each capable port to nic-offload, and nothing else changes. In this commit offload is still opt-in, behind DATAPLANE_NIC_OFFLOAD=1; "feat: offload whole peerings by default, with one switch to turn it off" later makes it the default and drops the variable for an annotation. - ports are probed with the driver arguments nic-offload asks for; - after configuration a port reserves template resources, and offloads nothing (with a warning) if that fails; - each such port gets a nic-offload-<port> thread that follows the published rate limits until shutdown, sending what the NIC lets through over the data queues with the port's RSS hash types. Tested on a BlueField-3 with an EVPN leaf stand-in and a 50 Mbit/s limit on one peering: a 300 Mbit/s flow was cut to 47.9 Mbit/s by the NIC (225,128 drops) with no over-limit traffic reaching software; traffic under the limit, the other peering, a live change of the limit, removing it, and shutdown all behaved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
mlx5 lowering becomes two steps: Plan::new turns a Program into data (pattern and actions templates, meter profiles, and the tables with their rules, each rule with its role), and install() creates exactly that on a port, deepest table first, and reports counts by role. A plan can be inspected and checked without the NIC; install is public so a tool can install a plan by itself. The NIC may only charge and drop what the software rate limiter would. Checking lowered plans against the software (a later commit) found four ways it did not: - Burst sizes were rounded down to avoid the sizes mlx5 encodes as zero, making the NIC stricter than software. Round up instead. - Software does not parse a VLAN-tagged frame inside VXLAN, so never charges one; the inner Ethernet match left it to mlx5 whether such a frame matches. Match untagged inner frames only (has_vlan = 0). - Software sees a fragment's protocol as "fragment", with no ports, so a rule naming a protocol never charges a later fragment; the NIC matched the IP header's protocol field. Rules naming a protocol match unfragmented packets only (IPv4 more-fragments and offset zero, IPv6 no fragment header). First fragments go to software too, which is safe: the NIC may always do less. - Software resolves extension headers to the protocol after them, so a rule for an extension header's protocol number (0, 43, 44, 51, 60) never matches; the NIC would match the field, and mlx5 refuses some outright, failing the whole port's offload. Refuse such rules (Refusal::ExtensionHeader). A pattern template's values are placeholders, but mlx5 validates them: protocol 0 is the IPv6 hop-by-hop extension header, and mlx5 refused such a table. Templates use UDP as the placeholder protocol. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
The software rate limiter becomes the oracle for the NIC. - model walks a mlx5 Plan over a frame: groups, table priorities, jumps, meters and their colors, matching patterns with dpdk::flow::model. It returns every outcome the NIC may produce: rules of one table keep no order, a meter may give any color, and a frame may be parsed in more than one way. - The differential test generates configurations (five VPCs of both families, four peerings, rules with subnets, ports and protocols, four meters including a chain and one that drops nothing) and frames (aimed at a rule or not, fragments, IPv6 hop-by-hop, inner VLAN tags, unroutable destinations). Every outcome for every frame is held to the cascade contract against what the software classifier does with the decapsulated packet: the NIC charges only the meter software charges, drops only red from such a meter, never misses in a group it jumped to, and every NIC meter is at least as permissive as the software component it stands for. One deviation is tolerated, explicitly, and pinned by a test: software charges no meter for a packet whose upper-layer protocol it cannot tell (a truncated transport header, an IPv4 protocol number of an IPv6 extension header), while a rule for any protocol charges it in the NIC. Which side should change is open. Reverting any of the previous commit's fixes, or the compiler's overlap refusal, fails the generated test within a few runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
install_probe lowers one program per rule shape (IPv4 and IPv6; any protocol, a protocol, a protocol and port; and all together) onto a port with hardware steering, installs and removes each, and reports what the NIC refused. The model cannot see a PMD's validation; this can, and it is where a qualification run for a NIC starts. On a BlueField-3 every shape installs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
mlx5 refuses some rule shapes outright. On a BlueField-3 an IPv6 rule with both prefixes longer than /96, both ports and a protocol fails "failed to validate pattern template" (/97 and /96 fit, /97 and /97 do not; /128 and /128 fit with one port): the match key does not fit. One such rule failed the whole plan, and the port's offload failed open to software. Now a rule of the program whose template, table or insertion the NIC refuses is left out and stays in software alone; the rest installs. That is always safe: rules charging different meters never overlap, so no remaining rule takes their traffic. Refusals of the root, the catch-all or a color group still fail the plan. - with_installed installs a plan, calls back while it is installed, and removes it; Installed reports the refused rules and the counters by role. install (hold until changed, report) is built on it and logs each refused rule. - Plan::outcomes_without models the plan as installed, without the refused rules. - install_probe probes the IPv6 prefix and port combinations and ICMP. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
The differential test holds the model to the software; qualify holds
a real NIC to the model. It is an ignored bolero test over the same
generators, run against two wired ports:
sudo NIC_OFFLOAD_QUALIFY=<port under test>,<sending port> \
<test binary> qualify --ignored --nocapture
Generated rate limits are compiled, lowered and installed on the port
under test (isolated, hardware steering); the generated VXLAN frames
are sent from the other port, each with its index in the outer IP
identification, which no rule matches. A frame arrives unchanged or
not at all, and must do what the model allows for the plan as
installed (without the rules the NIC refused).
Classification is checked without timing: every meter passes
everything, and in one phase per meter that meter's color group drops
every color, so what does not arrive is exactly what the meter
charged; the model walks the same rewired plan. Meters themselves are
not deterministic enough: on a BlueField-3 a meter passes a packet
while its bucket holds any tokens, and refuses an empty bucket. The
dropping rules' hardware counters must equal the frames meters
dropped. Most frames are aimed at a rule.
On a BlueField-3, 40 cases pass, including one where the NIC refused
two IPv6 rules.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
…settles Only 6 % of the frames sent in the dropping phases were charged by the meter being tested: frames were aimed at rules the compiler had not offloaded, and half carried an inner VLAN tag or were fragments, which no offloaded rule may charge. - Frames are aimed at configured rules the NIC was given, whole and untagged; one in eight stays as generated, and one in eight of the aimed ones keeps its fragment and VLAN tag, to check what the NIC must leave alone. - An aimed frame takes the ports of the subnets it is aimed into (the differential test gains from this too). - coverage, a model-only test, measures how many generated frames a meter charges and why the rest are not, and requires at least half: 62 % now, from 10 %. - Where the model allows a frame either to arrive or not, the runner tallies what the NIC did by the frame's features and reports it at the end. On a BlueField-3, 100 cases pass; 28 % of the frames in dropping phases are dropped (from 6 %). The only choice left open was IPv6 with a hop-by-hop header against rules naming a protocol: the NIC charged all 126, so mlx5 matches the protocol after the extension header, as the software does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
With today's semantics kept: the ACL filter and the rate limiter each
decide by their own first match, the filter first, and the NIC does
the same in two groups. A frame goes through the ACL denies, which
drop it, and then the rate limits:
group 0 VXLAN -> jump 1 (or 2)
group 1 ACL denies, by shape -> drop; last: jump 2 (or the fate)
group 2 rate limits, by shape -> meter / color group; last: fate
- The compiler takes the ACL filter's typed rules as well as the rate
limiter's, through one lowering for both (Match). A deny is
offloaded if no earlier rule of its table that allows overlaps it: a direct match is final in
the filter, whatever the flow state. Allows and default actions stay
in software (a default deny depends on flow state: replies). Every
refusal names its software rule (Origin).
- Rules the NIC refuses stay in software, denies as rate limits.
- The offload thread follows both the ACLs and the rate limits
(nic_offload::Policy), and rebuilds when either changes.
- The differential test generates per-peering ACLs (default action,
allow and deny rules, flow or packet scope) and checks the NIC
against the ACL filter, then the rate limiter: a NIC deny must be a
direct deny in software (a default deny is not enough: the reply of
an allowed flow passes). Disabling the earlier-allow check is
caught.
- The NIC may charge a meter for a packet software drops before its
rate limiter (denied, or unclassifiable): such traffic uses up the
NIC's budget only. This is tolerated explicitly and counted by
cause: with the denies in the NIC, what remains is nearly all
default deny.
- The qualification runner and its coverage test aim frames at ACL
denies too; install_probe probes every shape as a deny as well.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Offload had one switch, an environment variable. The dataplane's command line and environment cannot be changed in deployments, and offloading part of a peering's policy leaves edge cases, so: - mlx5 ports always start with hardware steering, and offload is attempted unless turned off. - gateway.alpha.githedgehog.com/attempt-offloads: "false" on any peering turns NIC offload off for the whole gateway (peerings are what carries annotations to the gateway); "false" anywhere wins. The validated rate-limit tables carry it, and RateLimits answers attempt_offloads(). - A peering is enforced in the NIC whole or not at all: every ACL deny and every rate limit it has, or none. A default deny (which depends on flow state), a deny an earlier allow overlaps, a rate limit another meter's rule could claim traffic from, or a match the NIC cannot express keeps the whole peering in software. Rules that drop nothing (allows, meters that drop nothing) need nothing in the NIC. Peerings never overlap (their VNIs differ), so each is decided alone. If the NIC refuses any rule of a peering at install, the plan is removed, and installed again without that peering. - No NIC rule matches a fragment: software sees a first fragment as its protocol and a later one as "fragment", which the NIC cannot tell apart alike for every rule. A deny that let a first fragment through to an any-protocol rate limit had it charged in the NIC only; now fragments are left to software whole. - The NIC model parses as mlx5 does (mlx5::PARSER: IPv6 extension headers walked, the protocol matched after them). With this, the differential test tolerates only one deviation: the NIC may charge a meter for a packet the ACL filter cannot classify. A NIC charge for a packet an ACL denies is a violation. The qualification runner installs as production does, taking out peerings the NIC refuses, and aims frames only at what the NIC took. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
The ACLs match TCP flags so that a crude defense against SYN floods can be configured (deny SYN without ACK), and a flood is where the NIC matters most. Rules matching flags used to be refused, keeping their whole peering in software; the NIC can match them, our wrappers just could not say so (TcpMatch::flags, an earlier commit). - Match carries TCP flags (values and mask), lowered for TCP rules into TCP-shaped tables; flags on another protocol stay refused. No fragment matches any NIC rule, so a later fragment, which has no flags in software, is left to it. - The differential test generates flag rules (packet scope, as the ACL filter requires) and frames carrying the flags of the rule they are aimed at; a SYN-without-ACK deny is checked explicitly (it drops a SYN in the NIC, not a SYN-ACK). install_probe probes SYN-without- ACK shapes; the qualification runner reports the flag rules it put in the NIC. On a BlueField-3 every flag shape installs, as a rate limit and as a deny, and 80 qualification cases pass, 18 of them with flag rules in the NIC (72 rules). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Every change of policy used to tear the NIC's rules down and build
them again, leaving the NIC enforcing nothing in between and resetting
every meter. Now each source VPC's rules are a generation of their
own, reached through its own rule in a switch group, and a port is
reconciled with a plan one VPC at a time:
root VXLAN -> jump switch (fixed)
switch one rule per source VNI -> its generation
last -> software
deny(v) the VPC's denies, by shape -> drop; last -> police(v)
police(v) its rate limits, by shape -> meter / color(m)
color(m) per meter -> count, pass or drop
- Plan names groups for what they are for (Group), numbered when
installed; jumps between them are per rule, so templates are shared
across generations. Generation tables no longer match the VNI (the
switch did), which frees enough of mlx5's match key for the IPv6
shapes it refused before. The model walks the same layout.
- mlx5::Port is the reconciler, on rekon's synchronous traits. What a
port holds is known from the handles of what was created, each made
once its creation completed (the shared template handles, which own
their dependencies). A VPC is attached by building its generation
aside, unreachable, then inserting its switch rule; detached by
removing its switch rule, then tearing the generation down; both
atomic for a packet. A VPC whose policy changed is detached, then
attached again, so software alone enforces it in between (switching
in place would leave the old policy in the NIC while software
enforces the new). Other VPCs, and meters whose key (name and
profile) did not change, with their token states, are left alone. A
generation the NIC refuses part of is never attached.
- If removing a rule fails, what the NIC holds is no longer known: the
root rule is removed, so software handles everything, and the port's
offload stays off until it restarts (Error::Poisoned, naming the
rule).
- mlx5 takes two identical rules into one table, but then fails to
remove the second, which would poison the port; generated policies
produce them easily (overlapping subnets of one rule, rules repeated
across entries). A rule identical to an earlier one of its source
VPC adds nothing (it charges the same meter, or drops the same), so
the compiler leaves it out, and so does the plan, whatever program
it is given. Prefixes are made canonical (host bits cleared) so
rules matching the same addresses compare equal.
- The offload thread keeps one Port and reconciles it with each new
policy; with_installed (probe, qualification) installs a plan on a
fresh Port.
On a BlueField-3 all 52 probed shapes install, including the IPv6
ones refused before, and 80 qualification cases pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
qualify_transitions generates sequences of policies, each step keeping some peerings' rules and ACLs from the step before, so that some source VPCs change and others do not. One port is reconciled from each policy to the next while the generated frames are sent in continuous cycles, each frame tagged with its cycle and index; meters pass everything, so what arrives depends on the denies: - cycles entirely before or after a reconciliation must do exactly what that policy's model says; - cycles overlapping one may lose a frame only if the old or the new policy denies it (the switched VPC is in software in between); - a meter both policies have keeps counting: it was left alone. reconcile_scenarios reconciles named sequences of plans on a fresh port without traffic; a port must never fail to remove what it installed (identical rules did). NIC_OFFLOAD_QUALIFY_QUIET (no traffic) and NIC_OFFLOAD_QUALIFY_LOG (the mlx5 PMD's log level) help diagnose a failure. On a BlueField-3: 20 sequences, 72 reconciliations, about 12,400 cycles checked exactly and 5,800 overlapping a reconciliation, with up to 5 VPCs attached: no disagreement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Review found ways the NIC could keep enforcing a policy software had moved on from: - A plan that failed to build (too many meters) left every VPC's old generation attached. Peerings now stay in software, the last meters' first, until the meters fit; and any other failure clears the port before reporting software-only. - A lost port exports nic_offload_poisoned. - Operations that time out, fail on completion, cannot push or pull their completion, or are still pending leave the NIC's state unknown: they lose the port, rather than counting as refusals. - Groups were handed out again whatever became of their tables. Rules and tables are now removed explicitly, and a group is handed out only once its tables are known gone. Meter color groups no longer leak on clear(). Errors are typed (Error::Lost with the operation and its TemplateError, Error::Poisoned, MeterNotMade, TableRefused, SwitchRefused), so a timeout is told apart from a refusal. The policy is compiled only from ACLs and rate limits seen together. The report timer reads the clock crate. Offload state is exported: nic_offload_vpcs, nic_offload_peerings_in_software. On a BlueField-3: reconcile_scenarios, 20 qualify cases and 8 transition sequences pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-Authored-By: Codex <noreply@openai.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
A rule's destination prefix stood in for its destination VPC, refused
only when it was every address. When a peer exposes a default or a
wide prefix, the prefix does not: a deny or rate limit of the peering
of VPCs A and B also matched A's traffic to a more specific prefix of
another peer C, which software sends to C. The NIC then dropped or
charged traffic software forwards.
Rules are now narrowed with the flow filter's destinations (Policy
gains its reader):
- a rule for peer B keeps, of its destination, only what overlaps one
of B's prefixes from the source VPC, one rule per prefix (none: the
rule matches nothing software sends B, and is left out);
- the other peers' prefixes at least as specific as the one a rule
was narrowed with pass to software ahead of every rule of the source
VPC (Program::passes), so their traffic skips the rule. Passing more
than they take (ports, protocols, gated prefixes) only leaves more
to software.
A rule of every address is no longer refused: narrowed, it is what
its peer takes. Identical rules are deduplicated after narrowing, so a
later rule for another peer is no longer dropped as identical to an
earlier one.
The differential test generates default, wide and overlapping exposes
(validated side by side), and routes frames by the flow filter's
destinations rather than its own idea of them. With narrowing disabled
it fails ("the NIC charges meter chain for a packet the software
denies"). ACL denies now count what they drop in the NIC
(acl_nic_dropped_packets_total).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
A NIC meter could turn red before the software meter it stands in front of, on traffic software would pass: - it counts the outer VXLAN frame where software bills the inner IP bytes, so small packets got about half the rate; - it passes a packet while any token is left and goes into debt, where software needs the whole cost; - software's per-worker leases let a burst exceed its bucket by up to a quarter; - mlx5 encodes rates to the nearest it can; - software couples its excess bucket to the committed one (MEF 10), the NIC's RFC 4115 does not. The NIC profile now scales byte rates and bursts by the most a packet's NIC count can exceed its bill ((20 + 72) / 20: an IPv4 header against the outer headers, an outer VLAN tag, the inner Ethernet header and the FCS), adds a quarter and one jumbo frame to each burst, adds 1/64 to rates, and programs the excess rate as the sum of both. With both components, only the bytes one is offloaded: software meters packets with the colors the bytes component left, which a NIC meter cannot. A timed property test runs one arrival sequence through the software meter and a model of the NIC's (outer bytes, debt, uncoupled or srTCM), and holds that traffic software has colored nothing red, the NIC colors nothing red either; without the margins it fails. No margin can make every NIC red a software red once software drops: it keeps the tokens of a packet it colors red, which the more permissive NIC spent. The scaling is for the smallest packets: those get through the NIC at about the rate software allows them, full-size ones at up to about 4.4 times it. The NIC stops floods, software enforces the rate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
…ails
NIC offload must never cost more than the offload. A NIC whose
firmware cannot do hardware steering refuses the probe with
dv_flow_en=2: the port is then probed as it would be without offload,
with a warning and nic_offload_probe_fallback{pci} set, and its NIC
offloads nothing (reserving template resources fails, as for any port
that cannot take them, and leaves the port running).
Not covered: a port that probes and configures under hardware steering
but then fails to start.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Software decapsulates only VXLAN addressed to the VTEP of the VNI's FIB; other VXLAN through the gateway is routed in the underlay untouched. The NIC matched the VNI alone, so it applied a VPC's inner denies and rate limits to transit VXLAN carrying the same VNI. Each VPC's switch rule now matches the outer IPv4 destination as well: its VTEP, which the policy reads from the FIBs (`Policy::vteps`). A peering whose VPCs lack an IPv4 VTEP stays in software. The VTEPs are part of what is seen together, so a VTEP change rebuilds the VPC's generation. The differential test checks that transit copies of every frame are neither dropped nor charged (a switch rule without the VTEP match is caught), and the hardware qualification sends them too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Offload is on by default, with an annotation to turn it off, not behind a flag with per-peering opt-in; the program has ACL denies too; a changed VPC is detached, then attached, not switched; the oracle is the ACL filter as well as the rate limiter. Also say the fragment rationale and the dedup rationale once each. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Fabric now passes the gateway's own annotations to the agent (GatewayAgentSpec.gateway.annotations). attempt-offloads is a gateway-wide switch, so it moves there from the peerings: a gateway without peerings can set it, and peerings can no longer disagree about it. - converters::k8s::config::offload reads it from the gateway, rejecting values other than "true"/"false" and unknown alpha gateway annotations. On a peering it is refused, with a pointer to the gateway. - It lives on the Overlay (disable_offloads, ValidatedOverlay:: offloads_disabled) rather than in the rate-limit tables. The rate limiter still publishes it with its rules (attempt_offloads()), so NIC offload compiles from one consistent snapshot. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Review found that rte_flow_flush under hardware steering also destroys the port's classic rules (mlx5_flow_list_flush), the control plane's steering among them, and that clear() took the root rule before lose() could see it, so a failed root removal skipped the fallback. - A lost port removes its root rule; if that fails, or a failed removal already took it, it removes every VPC's switch rule, so the root leads only to the switch's catch-all and software. Nothing else of the port is touched. Should that fail too, the port says so: Port::stale, exported as nic_offload_stale_policy. - When the policy changes while a program compiles, the VPCs that program changes are detached before compiling again: a policy republished faster than it compiles no longer leaves the NIC enforcing an old one. - acl_nic_dropped_packets_total and rate_limit_nic_packets_total keep the counts of removed rules, so they no longer go back on a rebuild. - nic_offload_peerings_in_software counts every routed peering with no rule in the NIC, so all of them while offload is off or failed. - A generation that runs out of groups gives back those it took. - FlowFilterContext::reference is infallible now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
…oftware Review found that narrowing took a gated (masquerade) or partial (port forwarding) destination as taking all of its prefix. It takes only replies, or only some protocols and ports: software sends the rest of the prefix to another peer, even a less specific one such as a default expose, while the NIC still applied the rule narrowed to it. A rule is no longer narrowed to such destinations, so software alone enforces it there. The differential test now generates port-forward and masquerade exposes, aims a quarter of frames at the host a port forward publishes, and routes each frame through the flow filter stage itself rather than a model of it. A directed test covers a masquerading peer next to a default expose (and a port-forwarding one); narrowing to gated and partial destinations again fails it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
…oding Review of the meter margins: - Rates gained 1/64 for mlx5's rounding to the nearest rate it encodes. They are now rounded up to the least it encodes exactly, from its own table (1e9 * m >> e), in its units (packet mode scaled by 2^7), so mlx5 programs at least that. A meter faster than the fastest rate it encodes stays in software with its peering (mlx5::unfit), as install and the differential test both apply. - The lease slack was a quarter of the burst. It is now what software bounds it by, one lease per worker: SharedMeter::lease_slack, through rate_limit's Component and the program's PoliceMeter. With policer's lease sizing that is at most a quarter of the burst but for its one-token floor, which holds back nothing a packet's cost does not take: the old figure was enough in practice, this one is the bound. - check_profiles compares what mlx5 encodes, with the lease slack, and the NIC meter model runs on encoded values. The property test meters software with the shared meter, packets arriving at up to 16 workers. Dropping the rate rounding fails check_profiles. - The doc had the scaling backwards: the smallest packets get through the NIC at about the rate software allows, full-size ones at up to about 4.4 times it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Keep the switch available during insertion so poisoning can remove existing switch rules when root removal fails. Remember unconfirmed root and switch operations to report stale policy even when no usable rule handle was returned. Exercise push and pull failures, root-removal fallback, failed switch removals, and clean refusals through the DPDK fault-injection fixture. Uncertain insertions must not recycle groups their rules could still reach. Co-Authored-By: Codex <noreply@openai.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Keep a peering with any logged ACL rule entirely in software, including its rate limits and both directions. This preserves logs for denies and for allows that a NIC meter could otherwise drop before the ACL filter sees them. Cover logged policies in the differential generator and directed IPv4/IPv6 tests. Keep hardware drop-coverage campaigns focused on offloadable policies; transition tests still generate logged ACLs. Co-Authored-By: Codex <noreply@openai.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
The offload thread polls the VTEPs every 200 ms to notice a change. It asked for one VNI at a time, twice per flow-filter destination, and each question made a new left-right read handle on the FIB table: thousands of handles registered and dropped per poll on a large gateway, competing with FIB publishing even when nothing changed. Vteps now answers for a set of VNIs with one table handle (the FIBs' handles are already cached per thread), and the poll asks for each VNI once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Compiling scanned every rule kept so far for each narrowed deny and rate limit, and Plan::vpc did the same per source VPC: about 10^8 Match comparisons at 10k rules, repeated on every recompile (homeless, unfit, meter cap, refusals) and on every port's thread. Both now keep a hash set of what they have seen. Narrowing also scanned every flow-filter destination, of every source VPC, for each rule and again for each of the rule's own destinations; the destinations are now grouped by source VPC once per compile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
A removed meter's counts were kept by name for the life of the port, so every name ever removed stayed in memory and in what report() exports every 10 s. They are now kept only while the policy still has a meter of that name (its totals still never go back); a name dropped and added again starts over, which a counter reads as a reset. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Port::args returned None both for a meter that was not made and for a
group its plan does not lay out, and the caller reported both as
MeterNotMade. It now returns a Result, with a new Error::NoSuchGroup for
the second, as the development guide's error handling asks ("Option is
not a good error").
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
reconcile made the plan's new meters while the meters no plan wanted were still installed, and removed those only at the end. A policy change that rekeyed many meters ran out of reserved meters part-way, and the peerings charging the ones not made stayed in software until the policy changed again. Unwanted meters now go first: no attached generation charges them, as an attached generation's spec, its meters with it, is the plan's. The reservation goes from 1024 to 65,536 meters a port (mlx5 takes up to 2^25 on a BlueField-3; each reserved one costs host and NIC memory from the start). Matching plan meters against installed ones is indexed by name, as a linear search per meter would be quadratic at that size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
When the NIC refused only rules that map to no program rule (a source VPC's pass or catch-all, or its switch rule), the refused peerings came out empty, which is a subset of any exclusions, so install_policy returned the whole program as installed although that VPC's generation was torn down. Nothing retried until the policy changed, and the gauge counted the VPC's peerings as in the NIC. Such a refusal now stands for every peering of the VPC's rules (a color rule's, for those charging its meter); one that stands for no peering turns the port's offload off until the policy changes, rather than installing the same plan again. nic_offload_peerings_in_software now counts from what is attached (each generation's spec records its peerings), not from the program last compiled: it was every peering after a policy change cut installing short, while unchanged VPCs stayed attached. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
profile() shifted a saturated packet-mode burst left by PPS_SHIFT without checking, so a large enough burst wrapped to a small NIC bucket, and unfit() checked rates only. mlx5 refuses bursts past MLX5_SRTCM_XBS_MAX (255 << 31, after its packet-mode shift), and encodable() returned sizes past 255 << 56 unrounded, in a band mlx5 encodes as zero. profile() now works in u128 and returns None for a rate or burst past what mlx5 takes, as a smaller one would be stricter than software; unfit() is whatever profile() refuses, so the two cannot disagree, and Plan::new returns the new Error::Unfit for such a meter. qualify's coverage cases leave unfit peerings out as run does: before, their rates were silently capped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
A source VPC's passes (other peers' destinations inside one of its rules' destinations) were laid out ahead of every rule of its entry group, with DELIVER. They also skipped the denies and rate limits of the peer owning the pass prefix: with A peering B on a default expose and C on 10.1.0.0/16, A's deny or rate limit for C was installed but never matched, while it counted as enforced in the NIC. The differential test could not tell, as delivering is always allowed. Rules now carry a level, the prefix length of the most specific of their peer's destinations they were narrowed to, and a group's tables are ordered by level, most specific first: of one length, rules before passes. A pass then follows the rules of its own destination and of more specific ones inside it, and precedes the rules of the less specific destinations it carves out of. Passes are in both groups; one in the deny group goes on to the police group, as the deny group's catch-all does. A rule is not narrowed to a destination another peer has too (which takes its traffic is flow-filter's order). Extra tables appear only where destinations nest. Subtracting the passes from each rule's prefix instead was considered: it needs a table per prefix length of the pieces, about 24 for a default expose minus /24s, where each table costs a lookup (~1.5 ns). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
What of the hugepages EAL was given the pools, rings, ACL tables and
drivers have allocated was not visible anywhere: EAL maps the whole
--numa-mem up front, and runs without telemetry.
- dpdk: SocketId::heap / socket::Manager::heap read
rte_malloc_get_socket_stats; Dev::heap gives the heap of a device's
node.
- The driver logs each NIC node's heap once the ports are up, and
publishes dpdk_heap_bytes{node,state=total|free|allocated} and
dpdk_heap_largest_free_bytes{node} every second.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Production gives the gateway far more 1 GiB hugepages than the default rings need. Init now uses them for deeper receive rings, and treats an allotment as a budget: - With --hugepages-gib N, N 1 GiB pages per NIC NUMA node are the budget. They must be available: there is no falling back to 2 MiB pages. The ports and workers must fit in them, or init fails with what they need. Workers are taken as given. - Interfaces without /rxd=N get the deepest of 4096, 2048 and 1024 receive descriptors per queue whose pools leave RESERVE_MIB (512 MiB) of the budget free, chosen per node and logged; init passes the choice to the dataplane in the launch configuration. Deeper than 4096 is never chosen. - Without --hugepages-gib, an interface without /rxd=N gets 4096 or 2048 only if 1 GiB pages already free hold that with the reserve; init does not grow the pools for a deeper ring. Otherwise 1024, with the requirement computed (and the pools grown) as before. - An explicit /rxd=N is never changed. The reserve covers what the heap holds beyond the pools: measured on a BF-3 with two ports and NIC offload, about 80 MiB plus 3.6 MiB per worker, and at most 99 MiB more for 16,384 ACL denies and 2,048 rate limits through a change of every rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
test_cli_handling took the reply to a ctl request as proof that the router had done everything asked of it. It had processed every ctl message, but it records the FRR config as applied only when the (fake) frr agent answers, which can come later: the check of the last applied config then failed now and then (2 of 15 runs locally, and in CI). The sync now polls until the config shows as applied, for up to 10 s, as frr/test.rs does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
…understood an_idle_wait_sleeps_until_its_deadline fails now and then on AMD, in CI and locally: the EAL hands MWAITX the low 32 bits of the absolute TSC deadline (DPDK 1213a8895b, still so upstream), where its timer takes the time to wait, so an idle wait ends at the next interrupt instead (2 us to 1 ms for a 20 us wait on a Zen 4 workstation). Passing MWAITX the time to wait made every wait 10 us on a Threadripper 7000 (family 0x19 model 0x18), but on the Ryzen 7000 CI runner (model 0x61) every such wait returned in about 450 ns, so that change is not made yet. The test is ignored with a note until this is understood, so CI does not fail at random, and a failure now reports the CPU's vendor, family and model, the sleeper, and every wait's duration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Daniel Noland <daniel@githedgehog.com>
Signed-off-by: github-actions[bot] <224724778+hedgehog-dataplane-update[bot]@users.noreply.github.com>
|
Important Review skippedBot user detected. To trigger a single review, invoke the
 |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
daniel-noland
force-pushed
the
hugepages/rxd-budget
branch
from
November 3, 2026 02:12
7b6a57a to
5324052
Compare
Base automatically changed from
hugepages/rxd-budget
to
offload/3-nic-offload
November 3, 2026 02:18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bump dataplane version to 0.37.0.
Triggered by @daniel-noland.
Do not forget to tag after merging!