Skip to content

feat: add queue and nic fields to metric_desc.names mapping - #188

Merged
atheurer merged 1 commit into
masterfrom
feat/add-queue-nic-names-fields
Jun 5, 2026
Merged

feat: add queue and nic fields to metric_desc.names mapping#188
atheurer merged 1 commit into
masterfrom
feat/add-queue-nic-names-fields

Conversation

@pradiptapks

Copy link
Copy Markdown
Contributor

Add two new keyword fields to the metric_desc.names mapping:

  • queue: for per-queue DPDK metric breakout (e.g., NIC hardware queues 0-N). Enables --breakout queue to query all queues in a single command instead of embedding the queue ID in the metric type name.

  • nic: for NIC PCI address identification. Enables --breakout nic to distinguish which physical NIC a port number corresponds to across different DPDK applications on different hosts.

These fields support the tool-dpdk telemetry tool's per-queue and per-NIC metric normalization. Without these fields, OpenSearch's strict dynamic mapping rejects documents containing them.

Add two new keyword fields to the metric_desc.names mapping:

- queue: for per-queue DPDK metric breakout (e.g., NIC hardware
  queues 0-N). Enables --breakout queue to query all queues in
  a single command instead of embedding the queue ID in the metric
  type name.

- nic: for NIC PCI address identification. Enables --breakout nic
  to distinguish which physical NIC a port number corresponds to
  across different DPDK applications on different hosts.

These fields support the tool-dpdk telemetry tool's per-queue and
per-NIC metric normalization. Without these fields, OpenSearch's
strict dynamic mapping rejects documents containing them.

Co-authored-by: Cursor <cursoragent@cursor.com>
@k-rister

k-rister commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Seems fine to me but @atheurer is the proper authority.

@pradiptapks

Copy link
Copy Markdown
Contributor Author

@atheurer @k-rister ,

This PR adds two new keyword fields (queue and nic) to the metric_desc.names mapping. These are needed by tool-dpdk, a new DPDK telemetry collection tool that collects per-port, per-queue NIC statistics during crucible benchmark runs.

Why these fields are needed:

Without queue and nic in the OpenSearch mapping, documents containing these labels are silently rejected by the strict dynamic mapping on metric_desc.names. This was confirmed during live testing -- all dpdk metric documents were dropped with:

strict_dynamic_mapping_exception: mapping set to strict,
dynamic introduction of [queue] within [metric_desc.names] is not allowed

What each field enables:

  1. queue (keyword) -- Per-NIC-hardware-queue breakout. DPDK NICs have multiple RX/TX queues (typically 4-16). Currently, the queue ID must be embedded in the metric type name (xstat-rx_q0_packets, xstat-rx_q1_packets, ...), requiring one CLI query per queue. With the queue field, all queues are returned in a single query:
# Before: 4 separate queries
crucible get metric --source dpdk --type queue-q0-packets --breakout hostname,port
crucible get metric --source dpdk --type queue-q1-packets --breakout hostname,port
crucible get metric --source dpdk --type queue-q2-packets --breakout hostname,port
crucible get metric --source dpdk --type queue-q3-packets --breakout hostname,port

# After: 1 query, all queues
crucible get metric --source dpdk --type xstat-q_packets --breakout hostname,port,direction,queue
  1. nic (keyword) -- NIC PCI address identification. DPDK port numbers (0, 1, 2...) are local to each application. Port 0 on TRex is a different physical NIC than Port 0 on OVS-DPDK. The nic field stores the PCI address (e.g., 0000.4b.00.0) so users can identify the physical NIC without cross-referencing external documentation:

crucible get metric --source dpdk --type rx-pps --breakout hostname,port,nic
Dependency:

tool-dpdk PR #3 depends on this PR. The tool-dpdk code already emits queue and nic labels -- they are accepted when the schema includes these fields and silently dropped when it doesn't. The tool-dpdk PR should be merged after this one.

Impact on existing tools:

None. These are additive keyword fields. No existing tool emits queue or nic labels, so no existing data or queries are affected. The names object is a flat map -- adding new fields does not change the behavior of fields already present.

Testing:

Verified on a live OVS-DPDK + testpmd + TRex topology (3 hosts, 13 DPDK ports) that:

  • Without this PR: dpdk documents with queue label are rejected (strict_dynamic_mapping_exception)
  • With this PR applied locally: documents are accepted and queryable via --breakout queue

@atheurer
atheurer merged commit 7629015 into master Jun 5, 2026
35 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in Crucible Tracking Jun 5, 2026
@pradiptapks
pradiptapks deleted the feat/add-queue-nic-names-fields branch June 8, 2026 02:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants