Skip to content

feat: register explicit shared-lN-domain fields for CPU cache breakouts - #208

Merged
k-rister merged 2 commits into
masterfrom
feat-cpu-cache-domain-schema
Aug 19, 2026
Merged

feat: register explicit shared-lN-domain fields for CPU cache breakouts#208
k-rister merged 2 commits into
masterfrom
feat-cpu-cache-domain-schema

Conversation

@k-rister

Copy link
Copy Markdown
Contributor

Summary

  • Registers shared-l1-domain through shared-l4-domain as explicit keyword fields in metric_desc.base, to support tool-sysstat's upcoming per-cache-level breakout (issue mpstat: add 'node' (NUMA node) and 'cpu' (alias for num) as CDM breakout dimensions tool-sysstat#68).
  • Corrects CLAUDE.md: a dynamic_templates wildcard approach was tried first but verified live against OpenSearch to provide no exception under "dynamic": "strict" — a field matching a dynamic_templates glob is rejected exactly like any other unregistered field. Explicit fields are the only working option and match this schema's existing convention.

Test plan

  • Built metric_desc.json from the .base fragments via build.sh/Makefile, confirmed valid JSON with the new fields present and no dynamic_templates key
  • Loaded the built template into a live OpenSearch instance under a throwaway index name; confirmed a document with shared-l3-domain is accepted and a document with an unregistered shared-l5-domain is rejected with strict_dynamic_mapping_exception

Registers shared-l1-domain through shared-l4-domain as explicit keyword
fields in metric_desc.base, for tool-sysstat's upcoming per-cache-level
NUMA-style breakout (mpstat cpu utilization grouped by shared cache
domain).

A dynamic_templates wildcard was tried first but verified live against
OpenSearch to not work -- "dynamic": "strict" rejects any field
matching a dynamic_templates glob exactly the same as an unregistered
field; there is no bypass. Explicit fields match this schema's existing
convention and fail loudly (one rejected document, not silently lost
data) if a future host reports an L5+ cache level.
@k-rister k-rister self-assigned this Aug 18, 2026
@k-rister
k-rister requested a review from a team August 18, 2026 15:29
@project-crucible-tracking project-crucible-tracking Bot moved this to In Progress in Crucible Tracking Aug 18, 2026
@k-rister

Copy link
Copy Markdown
Contributor Author

PR Review: CommonDataModel#208 — feat: register explicit shared-lN-domain fields for CPU cache breakouts

Summary: Registers shared-l1-domain through shared-l4-domain as explicit keyword fields in the metric_desc index mappings to support per-cache-level CPU breakouts from tool-sysstat.
Changed files: 2
Review dimensions: Correctness, API & Contracts, Build & Deploy, Documentation, Style, Completeness


Issues

  • [queries/cdmq/cdm.js] Programmatic index schema mappings are missing the new CPU cache level breakout fields
    • Description: queries/cdmq/cdm.js defines the programmatic schema mappings for Elasticsearch/OpenSearch indices in indexDefs['v8dev']['metric_desc'], which is subsequently deep-cloned for versions v9dev and v10dev (cloned to v10dev which is the current VERSION). The newly introduced fields—shared-l1-domain, shared-l2-domain, shared-l3-domain, and shared-l4-domain—are registered in templates/metric_desc.base but are entirely missing from queries/cdmq/cdm.js.
    • Triggering Scenario: If an index or its template is created or updated programmatically using the Node.js library (cdm.js) or via scripts like queries/cdmq/create-index.js, the resulting OpenSearch schema will be missing these four cache breakout fields. When tool-sysstat is run on a platform that emits these breakouts, the strict dynamic mapping exception ("dynamic": "strict") will be triggered in OpenSearch, causing indexing failure and data loss.
    • Action: Update the indexDefs['v8dev']['metric_desc'] breakout properties in queries/cdmq/cdm.js to register the same four fields under properties.names.properties.

File Coverage

  • templates/metric_desc.baseNo issues found
    • Successfully registers shared-l1-domain .. shared-l4-domain keyword fields under the names block, matching existing CPU hardware topology schema conventions.
  • CLAUDE.mdNo issues found
    • Documents the "dynamic": "strict" behavior of CDM indices, the failure of wildcard dynamic_templates, and the rationale for why explicit registration is required.

Missing from diff

  • queries/cdmq/cdm.js — Must register shared-l1-domain through shared-l4-domain under indexDefs['v8dev']['metric_desc'].mappings.properties.metric_desc.properties.names.properties to ensure programmatic schema creations stay in sync with the index templates.
diff --git a/queries/cdmq/cdm.js b/queries/cdmq/cdm.js
index cdf59e8..b9708ab 100644
--- a/queries/cdmq/cdm.js
+++ b/queries/cdmq/cdm.js
@@ -278,6 +278,10 @@ indexDefs['v8dev']['metric_desc']['mappings']['properties']['metric_desc'] = {
         mode: { type: 'keyword' },
         socket: { type: 'keyword' },
         domain: { type: 'keyword' },
+        'shared-l1-domain': { type: 'keyword' },
+        'shared-l2-domain': { type: 'keyword' },
+        'shared-l3-domain': { type: 'keyword' },
+        'shared-l4-domain': { type: 'keyword' },
         cluster: { type: 'keyword' },
         container: { type: 'keyword' },
         cgroup: { type: 'keyword' },

Limitations

  • Cannot verify real-time ingestion behavior on a target host without a running OpenSearch/Valkey stack and containerized executor, but static verification, local template compilation (build.sh/Makefile), and code trace confirm compatibility and correctness under normal operations.

Verdict

Request changes — The templates are correct and compile perfectly. However, the programmatic schemas defined in queries/cdmq/cdm.js are not in sync with the templates. This must be addressed to avoid indexing failures and strict_dynamic_mapping_exception crashes when using programmatic index creation tools or Node-based services.

queries/cdmq/cdm.js maintains its own copy of the metric_desc.names
schema (indexDefs['v8dev']['metric_desc'], deep-cloned into v9dev and
v10dev), separate from templates/metric_desc.base. It's load-bearing
in two places: checkCreateIndex()/updateIndexMappings() PUT it
directly to OpenSearch as an alternate index-creation path, and
add-run.js -- the actual production indexing script -- validates every
document's fields against it, rejecting anything not listed. Without
this fix, add-run.js would reject tool-sysstat's cache-domain data
before it ever reached OpenSearch, even with the template already
updated.

Found via PR review; verified by tracing add-run.js's field
validation and confirming v10dev inherits the fix through the
deep-clone chain.
@k-rister

Copy link
Copy Markdown
Contributor Author

Good catch — verified and fixed in cec90a0.

queries/cdmq/cdm.js does maintain an independent copy of this schema (indexDefs['v8dev']['metric_desc'], deep-cloned into v9dev/v10dev), and it's genuinely load-bearing: checkCreateIndex()/updateIndexMappings() PUT it straight to OpenSearch as an alternate index-creation path, and add-run.js (the actual production indexing script) validates every document's fields against it, rejecting anything not listed with "These fields must be added to indexDefs in cdm.js before indexing." Without this fix, add-run.js would have rejected tool-sysstat's cache-domain data before it ever reached OpenSearch, even with the template already updated.

Added the same four fields at the matching location in cdm.js, confirmed indexDefs.v10dev.metric_desc picks them up through the deep-clone chain (node -e check), and confirmed Prettier-clean.

@atheurer atheurer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Approved after compiling templates and validating the JSON schemas.

@k-rister
k-rister merged commit 1e733bf into master Aug 19, 2026
67 of 69 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in Crucible Tracking Aug 19, 2026
@k-rister
k-rister deleted the feat-cpu-cache-domain-schema branch August 19, 2026 00:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants