RFC: vswitch design

[JDuchniewicz]

Author(s)	@JDuchniewicz, @alexandermbrown
Date of last update	13 Feb 2026

Original PR with description and design: #177

Overview

The vswitch models a physical Ethernet switch where we have the ability to send and receive packets to all PDs connected to the vswitch. vswitch has to connect to at least one virtualiser TX/RX pair (further connected to a Device) and one or more clients - each consisting of a set of sDDF network queues (TX/RX active & free).

Each pair of TX/RX queues comprises a port, similar to Ethernet ports. Whether a given port can talk to another one is established by the routing table. How we set it up is one of the open questions of the design.

Filtering

Each port has an allow-list that specify which other ports on the vswitch it can access. The allow-list construction is done statically during the system file creation. We could allow for dynamic modification of that allowlist during runtime via a PPC.

Copying

Right now we are copying every packet that is being forwarded to avoid the problem of all free buffers accumulating on one client. When connecting the clients we don't need a copier component on the RX path anymore because the vswitch will handle that.

An open question is whether we can avoid the copy on the TX path, as it is being done in the systems without the vswitch on the TX path (from the client's perspective).

Example System

(Image courtesy of @alexandermbrown)

Open questions

Carrying over some questions from the original PR:

• Who should be doing the multiplexing: vswitch or the virtualiser? Currently virtualiser performs the multiplexing between all connected PDs and allows for broadcasting the packets to all of them. This means overlapping responsibilities between vswitch and the RX virtualiser.
  In the original PR there is also the idea of merging components but I think that's beyond scope

Some of my own:

• Can VSwitch have more virtualiser pairs connected? Does it even make sense?
• Do we want to allow for vswitch daisy-chaining? In such case we must resort to dynamic routing as we will have multiple MAC addresses corresponding to one port
• Can we avoid copying? Seems like we could get away with not copying on TX path from the virtualiser but that would require a good way of handling the buffers.

Option 1: Static routing table

In this option we have just a single virtualiser connected and multiple clients where none of them can be a vswitch
(they actually could but we would need to recursively find all the MACs that present themselves behind this port, then we have also the issue of having a virtualiser behind a vswitch connected to a vswitch).

We construct a MAC to client's port mapping during sdf creation. When the vswitch is operational and it receives a packet, it routes it to a known client using the static MAC to client mapping. If the destination MAC is unknown it means it has to be routed to the outside via the virtualiser port (to the device and to the outside world).

Option 2: Dynamic routing table

This is the design from the original PR. The routing table is constructed dynamically as the vswitch receives packets. The MAC to client mappings are then created and reused, but this still has to search through all existing mappings to see if we don't have a mapping to this MAC already which is slow. This has the advantage that we can have many various devices connected, possibly daisy-chained vswitches. But we have to trade-off some time during packet processing for the table creation and updating.

The original PR did not foresee vswitch behind a vswitch as there is still a one-to-one MAC and port number mapping (unless I am understanding the code wrongly).

Related branches

sddf
meta.py virtio_vswitch example
sdf changes

[wom-bat]

This is very very similar to what the routing components in the LionsOS firewall example have to do. It'd be really good to look at whether the routing and filtering components we already have can be generalised to multiple internal components as well as multiple external interfaces.

[JDuchniewicz]

After the discussion with @Courtney3141 we came up with some new ideas regarding the copy queue placement and queue management in general. The main conclusion is that the RX copy queue will still be a component connected to clients. We also decided that static routing table would be the best way to go forward.

From the architectural standpoint, for discussion's purposes we will assume there is just a single vswitch that has 2 clients (from here on named Client0 and Client1) connected to it and the vswitch itself is connected to the virt's (device). There is also a client (Client2) that is not connected to the vswitch but shares the virts(device). This is similar to the figure above but with additional client below the vswitch that is not connected to it.

(There are few other architectural options that we will not consider for now, but we talked briefly that they could work just fine:

• 2 vswitches talking to each other connected to different devices (possibly running on different cores)
• nested vswitches (should such case be ever created though?)
• 2 devices connected to one vswitch)

---

There are a few types of communication which would be happening:

Case 1: Packet is received by the driver and passed to the rxvirt (Client0 Rx)

• rxvirt knows only about Client2's MAC, rest is catchall - puts the buffer to the vswitch active queue (either presenting as a regular client queue or special vswitch one?)
• vswitch dequeues the buffer from the rx virt's active queue, looks up the receiver MAC and decides it should pass it to Client0, enqueues it in the Client0's Copier's active queue
• Client0's Copier's active queue copies the packet - the buffer is returned to the virt's rx vswitch free queue

Case 2: Client0 sends the packet for Client1 (Client0 Tx/Client1 Rx) - the same as above

• vswitch gets notified by the Client0, dequeues the buffer from Client0's active queue, returns free buffers
• vswitch looks up the receiver MAC, decides that it should pass the packet over to Client1, enqueues it in the Client1's Copier's active queue, returns the free buffers
• Client1's Copiers active queue copies the packet - the buffer is returned to the Client0 free queue

IMPORTANT: We have to copy in the Copier to enforce security as Client1 could modify Client0's buffer.

Case 3: Client0 sends the packet for outside world (not sure whether we want to have this) (Client0 Tx)

• vswitch gets notified by the Client0, dequeues the buffer from Client0's active queue
• vswitch dequeues the buffer, looks up the receiver MAC, since the dest MAC is not inside the vswitch routing table, the packet is placed in the virt tx active queue (if we have multiple devices this is more complex)
• virt tx is notified, reads the active queue, passes the data to the driver, returns the free buffers
• vswitch returns the buffer to the Client0 later

Case 4: Client0 broadcasts the packet

• vswitch gets notified by the Client0, dequeues the buffer from Client0's active queue
• we track the number of references to the buffer that will be created
• for every receiver copying it we decrement the reference count
• if the reference count drops to 0 we return the buffer to the free queue of Client0

I might need some help in clarifying the numerous buffer handling cases we have here @Courtney3141. We also discussed the concept of having a queue that contains free buffers from different clients where we would distinguish the client to return the buffer to basing on the id encoded in the length field of the buffer.

[Courtney3141]

rxvirt knows only about Client2's MAC, rest is catchall - puts the buffer to the vswitch queue

Catchall may optionally be replaced by virt having knowledge of all client's MACs, but client0 and client1 are passed to the same PD

Client0's copy queue copies the packet - the buffer is returned to the virt's vswitch queue

May possibly need to re-visit whether it can go directly to virt or via vswitch when we introduce broadcast/buffer refcounts

there is no copy, since we pass the packet in-vswitch (is that true?)

No copy pre vswitch yes

Client1 returns the buffer to the vswitch that will later be returned by it to Client0

client1 copy returns buffer (needs to be trusted)

Case 3: Client0 sends the packet for outside world (not sure whether we want to have this)

I thought this is a standard case?

does it need to be copied?

No, not necessarily - it is only handled by trusted components, so can be trusted to be returned correctly.

Probably at least one missed case of broadcast from the external world, which will need refcounts in at least 2 components I think (rx virt and vswitch). But I think it's all doable.

Buffer data pool ID numbers can be used as an alternative to multiple queues, in the case where we would like to enqueue buffers from multiple pools into the same queue (so we know where to return them/where to find them in our address space). We also get this for free with no extra memory with our current buffer descriptor struct.

[JDuchniewicz]

Probably at least one missed case of broadcast from the external world, which will need refcounts in at least 2 components I think (rx virt and vswitch). But I think it's all doable.

Take a look at updated comment - more detailed and I clarified that Case 1 and 2 are actually the same (except we could avoid the copy between clients connected to the same vswitch).

Buffer data pool ID numbers can be used as an alternative to multiple queues, in the case where we would like to enqueue buffers from multiple pools into the same queue (so we know where to return them/where to find them in our address space). We also get this for free with no extra memory with our current buffer descriptor struct.

Also, regarding broadcast from the external world: virt rx already handles that and needs implementation that will cover when vswitch finishes broadcasting to it's clients. If the virt does not know the client IDs that will receive these packets then we cannot really tag them and returned tagged packets back to virt. I think we should discuss this case more in-detail.

[Courtney3141]

I will re-read, but I have sketched out in more detail how I think broadcast should work, I will upload a diagram and notes. Currently I don't see any issues, but I could be missing something :-)

[Courtney3141]

[Courtney3141]

Case 1 issue:

vswitch dequeues the buffer from the rx virt's active queue, looks up the receiver MAC and decides it should pass it to Client0, enqueues it in the Client0's Copier's active queue, returns the free buffers

There is no free buffer yet. It is still active. So this point should end after enqueueing in the copier's active queue.

Client0's Copier's active queue copies the packet - the buffer is returned to the virt's rx vswitch free queue

After the copier finishes copying, it enqueues the free buffer in the free queue it shares with the vswitch.

Then, the vswitch processes the free queue it shares with the copier, and by checking the region ID associated with the buffer, it knows it belongs to the Rx DMA region so it enqueues it in the Rx virtualiser's free queue. Note that this region ID can be set by the vswitch when the buffer was enqueued by the vswitch, as it knows it is from the Rx DMA region as it came from the Rx virtualiser. The copier is a trusted component, so we can assume that it will not erroneously modify this ID.

[Courtney3141]

Case 2:

Yes, we can get away with the extra copy (I think).

• vswitch gets notified by Client0, dequeues the buffer from Client0 active queue thus knows which data region it belongs to and can tag it later.
• vswitch looks up the receiver MAC, decides that it should pass the packet over to Client1, enqueues it in the Client1's Copier's active queue and tags it with client0's data region.
• Client1's Copiers active queue copies the packet - the buffer is then enqueued in the free queue shared between client1 copier and vswitch
• vswitch awakens and processes client1's copier's free queue. Checks the buffer tag, finds it's client0's and returns it to client0's tx free queue.

[Courtney3141]

Case 3:

vswitch gets notified by the Client0 Copier, dequeues the buffer from Client0 Copier's active queue, returns free buffers

No copier needed here, client0 and vswitch interact directly. client0's buffer is not returned until DMA has been completed (and it has been passed back through other net components).

vswitch dequeues the buffer, looks up the receiver MAC, since the dest MAC is not inside the vswitch routing table, the packet is placed in the virt tx active queue (if we have multiple devices this is more complex)

• Let's ignore multiple devices for now...
• vswitch also tags buffer here with client0 data region

virt tx is notified, reads the active queue, passes the data to the driver, returns the free buffers

• virt tx will need to keep some booking of region ID --> physical address. Currently this is done by having each client have a different queue, and Tx virt knows phys ID of each client's data region. This can basically remain fairly similar, but since it is receiving multiple client's buffers in the same queue now we will have to make this slightly more explicit rather than implicit.

vswitch returns the buffer to the Client0's Copier later

Identifies the buffer is client 0's using the region ID. Also, returns to client directly, no copier.

[Courtney3141]

Case 4 should be covered by my pictures if you can read my handwriting

[JDuchniewicz]

Thanks for all of this! I yet again corrected the comment as I mistakenly insterted Copiers everywhere for Clients instead of just on the Rx path.

The idea of tagging the buffers should work just fine. Same with reference counting for broadcasts - I see it works already for broadcasting from virt Rx.

Re Case 2: (I talked with @cazb2 about that case)
Since we are not engaging any HW in the process, we could just be passing the buffer from Client0 to Client1 and bypassing the Copier (by using some special tag in the Copier that would tell it that this buffer's origin is intra-system and can be safely passed through). I think you still talk about regular case it in the comment #636 (comment) @Courtney3141
Clarified offline - we have to copy the buffer because we cannot trust the other Client to not modify the memory owned by some other Client.

Since we are not considering multiple devices connected to the vswitch, there is only one port that is different from others. My question is whether we should have special handling for this case (in the vswitch) and also in sdf. If not we could abstract away everything as was done initially under a port and have the common logic take care of "distinguishing" whether a given port is a Client or a driver.